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FOREWORD 


Historically,  Army  acquisition  research  has  had  difficulty  conducting  an  adequate  early 
assessment  of  the  human  dimension  in  system  performance.  Human  performance  research  is 
critical  to  Future  Combat  Systems  (FCS)  because  enhanced  battle  command  through  advanced 
command  and  control  (C^)  systems  is  at  the  heart  of  the  FCS  concept.  The  FCS  C^  program 
reflects  the  proactive  research  on  human  performance  needed  to  build  the  force  of  the  future. 
Attention  to  the  human  dimension  underscores  the  role  performed  by  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  as  the  Army’s  primary  research 
organization  for  Personnel,  Training,  and  Leader  Development. 

This  report  describes  and  documents  ARI’s  work  and  products,  particularly  measurement 
methods  and  results,  as  a  key  member  of  the  Human  Performance  Team  for  the  FCS  C^  program. 
The  work  reported  here  focused  on  measuring  human  performance  to  understand  and  address 
task  and  training  requirements  for  command  groups  in  future  FCS  organizations,  jhis  report 
provides  exemplar  methods  and  results  related  to  FCS  C2  Experiment  3  that  attend  to  the  tiuman 
dimension  of  battle  command  in  the  future  force. 

The  research  reported  here  reflects  ongoing  work  to  address  human  performance  issues 
by  ARI,  and  especially  the  Future  Battlefield  Conditions  (FBC)  Team  of  the  Armored  Forces 
Research  Unit  (AFRU).  This  report  supports  work  package  (211)  FUTURETRAIN;  Techniques 
and  Tools  for  Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance  (C^ISR)  Training  of  Future  Brigade  Combat  Team  Commanders  and  Staffs,  and 
supports  the  Science  and  Technology  Objective  (STO)  “Methods  and  Measures  of  Commander- 
Centric  Training.” 

Findings  from  this  effort  were  briefed  to  the  Deputy  Chief  of  Staff  for  Operations  and 
Training  (DCSOPS&T)  from  the  Training  and  Doctrine  Command  (TRADOC).  Methods  and 
findings  from  Experiment  3  were  provided  to  the  Program  Manager  (PM)  for  FCS  C^  as  part  of 
ARI’s  ongoing  support  to  FCS  and  Army  research  and  development  efforts. 
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FUTURE  COMBAT  SYSTEMS  COMMAND  AND  CONTROL  (FCS  a)  HUMAN 
FUNCTIONS  ASSESSMENT:  INTERIM  REPORT  -  EXPERIMENT  3 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army’s  challenging  transformation  to  Future  Combat  Systems  (FCS)  entails  an 
unprecedented  amalgam  of  humans  and  machines,  a  truly  hybrid  future  force.  A  pivotal  example 
of  the  FCS  transformation  challenge  is  the  requirement  that  a  relatively  small  command  group 
must  be  able  to  command  and  control  (C^)  an  interdependent  mix  of  manned  and  autonomous 
systems.  An  ongoing  research  program  called  FCS  C^  exemplifies  the  Army’s  effort  to 
proactively  meet  this  requirement.  To  ensure  a  focus  on  human  performance,  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  participates  in  the  FCS  C^ 
research  program.  This  report  provides  exemplar  research  methods  and  findings  on  human 
performance  by  ARI  for  Experiment  3  in  the  FCS  C^  research  program. 

Procedure: 

Primary  participants  were  Active  Duty  Lieutenant  Colonels  who  formed  a  notional 
FCS  command  group  to  more  fully  explore  and  develop  new  command  and  control  paradigms. 
Experiment  3  lasted  two  weeks  with  the  first  three  days  dedicated  to  training  and  the  remaining 
days  to  experimental  exercises  referred  to  as  “runs.”  The  execution  phase  of  each  run  lasted 
approximately  60-90  minutes,  and  was  proceeded  by  a  planning  phase.  Overall,  1 1  of  the  12 
runs  scheduled  for  Experiment  3  were  completed  and  analyzed  for  ARI’s  assessment  of  human 
performance. 

Efforts  by  ARI  in  support  of  training  and  evaluation  resulted  in  the  respective  use  of 
deliberate  practice  complexity  levels.  Design  for  deliberate  practice  stressed  the 

repetition  of  similar  runs  with  feedback  to  ensure  results  were  based  on  proficient  performance. 
Run  complexity  was  varied  (Medium,  High,  and  Too  High)  to  assess  how  changes  in  operational 
conditions  might  impact  command  group  performance,  and  to  gauge  the  performance  limits  of 
the  proposed  Unit  Cell  organization. 

The  setting  was  simulated  desert  terrain  from  the  National  Training  Center  (NTC)  in 
which  the  Unit  Cell  conducted  deliberate  attack  missions  against  a  battalion  (minus/plus)  to  clear 
passage  lanes  for  a  follow-on  force.  The  performance  feedback  essential  to  deliberate  practice 
included  end-of-day  AARs.  Design  goals  were  to  help  participants  learn,  assess,  and  refine  the 
new  technical  skills  required  to  operate  their  C^  prototypes  and  the  new  tactical  skills  required  to 
exploit  the  Unit  Cell’s  progressively  automated  assets. 


Measurement  methods  for  assessing  human  performance  were  developed  and  iteratively 
refined  by  ARI  across  the  PCS  experiments  to  better  understand  command  group  performance 
and  to  identify  training  requirements.  By  Experiment  3,  the  measurement  methods  designed  and 
developed  by  ARI  resulted  in  a  relatively  reliable  and  comprehensive  set  of  human  performance 
measures  that  are  fully  documented  in  this  report.  In  fact,  a  rationale  for  publishing  this  Interim 
Report  as  an  ARI  Research  Report  was  to  document  and  transfer  these  human  performance 
measurement  methods  to  future  research  efforts. 

Subjective  measures  of  human  performance  obtained  participant  feedback  on  multiple 
research  issues  including:  training,  skill  proficiency,  workload,  performance  success,  teamwork 
skills,  decision  making,  function  and  task  allocations,  prototype  effectiveness,  and  human-system 
integration. 

Objective  measures  obtained  detailed  and  comprehensive  data  on  the  verbal  and  human- 
computer  interactions  (HCI)  performed  by  the  command  group  participants  during  run  planning 
and  execution  phases.  For  selected  runs,  this  analysis  included  every  human-to-human  verbal 
interaction  and  every  human-to-computer  manual  interaction  performed  by  each  member  of  the 
command  group.  Analyses  related  the  relatively  micro  objective  behaviors  measured  to  more 
meaningful  functions  including  Plan,  See,  Move,  and  Strike. 

Automated  measures  of  performance  are  needed  to  improve  training  and  evaluation 
efforts.  For  Experiment  3,  ARI  requested  a  select  set  of  key  automated  measures  be  developed 
to  help  assess  command  group  interactions  with  their  prototypes.  Only  a  small  subset  of  the 
automated  measures  requested  was  developed,  however,  ARJ’s  efforts  to  validate  these  measures 
by  comparing  them  with  manual  measures  obtained  from  HCI  analysis  are  reported. 

Findings: 

The  human  performance  findings  reported  are  based  on  subjective,  objective,  and 
automated  measures  used  by  ARI  to  assess  command  group  performance  for  Experiment  3 . 
Overall,  the  body  of  subjective  and  particularly  objective  results  obtained  on  human  performance 
represents  an  emerging  empirical  database  on  command  group  task  and  training  requirements  in 
small  FCS  units. 

The  measurement  methods  developed  by  ARI  resulted  in  reliable  and  meaningful  data  on 
humans  performing  command  and  control  in  a  notional  FCS  organization.  Such  human 
performance  data  is  needed  to  understand  and  improve  command  group  performance,  and 
particularly  to  address  training  issues  including:  task  analysis,  task  allocation,  workload, 
performance  assessment,  and  training  requirements.  The  evaluation  framework  helped  relate 
relatively  micro  objective  behaviors  measured  to  more  meaningful  C^  functions  including  Plan, 
See,  Move,  and  Strike.  Results  on  development  and  validation  of  automated  measures,  however, 
were  meager  and  underscore  the  unmet  requirement  to  instrument  prototype  and  fielded  C 
systems  for  more  effective  and  efficient  training  and  evaluation. 

The  interim  conclusions  provided  focus  primarily  on  workload,  training,  and  human- 
system  integration  issues  related  to  FCS  command  and  control  for  small  command  groups.  In 
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closing,  a  small  set  of  sustain  and  improve  recommendations  are  provided  for  future  research 
efforts.  Notably,  more  comprehensive  research  methods  and  findings  on  future  command  group 
performance  are  provided  in  a  companion  report  that  addresses  PCS  Experiments  1-4 
(Lickteig,  Sanders,  Durlach,  Lussier,  &  Carnahan,  In  Preparation). 

Utilization  of  Findings: 

Methods  and  findings  on  human  performance  from  each  experiment  were  provided  to  the 
Program  Manager  (PM)  PCS  as  part  of  API’s  ongoing  support  of  PCS  and  Army  research  and 
development  efforts.  Findings  by  ARI  were  briefed  to  the  Deputy  Chief  of  Staff  for  Operations 
and  Training  (DCSOPS&T)  from  the  Training  and  Doctrine  Command  (TRADOC).  The 
measurement  methods  developed  by  ARI  to  measure,  analyze,  and  report  human  performance 
were  documented  to  facilitate  their  transfer  to  future  efforts,  particularly  research  on  battle 
command.  Training  issues  identified  should  guide  API’s  Science  and  Technology  Objective 
(STO)  titled  “Methods  and  Measures  of  Commander-Centric  Training”  and  future  PCS  training 
development  efforts. 
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FUTURE  COMBAT  SYSTEMS  COMMAND  AND  CONTROL  (FCS  C^) 

HUMAN  FUNCTIONS  ASSESSMENT:  INTERIM  REPORT  -  EXPERIMENT  3 

Introduction 

This  document  was  originally  prepared  as  an  interim  report  and  provided  to  the  Program 
Manager  (PM)  of  the  Future  Combat  Systems  Command  and  Control  (FCS  C^)  program  upon 
completion  of  Experiment  3.  Publication  as  an  ARI  Research  Report  provides  exemplar  research 
methods  and  findings  for  transfer  to  future  research  efforts.  To  more  clearly  indicate  the  human 
performance  documentation  available  from  PM  FCS  C^  in  ARI’s  interim  reports  for  Experiments 
1-4,  only  minor  editorial  changes  were  made  in  the  publication  of  this  Experiment  3  report. 
Notably,  more  comprehensive  research  methods  and  findings  on  future  command  group 
performance  are  provided  in  a  companion  report  that  addresses  FCS  C^  Experiments  1  -4 
(Lickteig,  Sanders,  Durlach,  Lussier  &  Carnahan,  In  Preparation). 

Purpose 

The  FCS  C^  program  is  a  joint  effort  led  by  the  Defense  Advanced  Research  Projects 
Agencys  (DARPA)®  and  U.S.  Army  Communications-Electronics  Command  (CECOM) 
Research  and  Development  Center  (RDEC).  During  October  2001  to  March  2003,  an  iterative 
series  of  command-in-the-loop  experiments  were  conducted  at  Fort  Monmouth.  As  a 
participating  member  in  this  effort,  ARI  serves  primarily  on  the  FCS  C^  Human  Performance 
Team. 


The  stated  purpose  of  the  FCS  C^  program  is  to  test  the  hypothesis  that  digitization  of 
current  battlefield  operating  systems  enables  a  new  approach  to  command  and  control: 

If  digitization  of  current  battlefield  operating  systems  can  substantially  enhance 
command  and  control  by  providing  better,  more  accurate,  and  timely  battlefield  data  to 
today’s  commander  and  staff  for  decision  making;  then  a  “new”  approach  to  Battle 
Command  and  Control,  implemented  in  the  form  of  synthesized/analyzed  information 
presented  to  the  future  Unit  Cell  Commander,  will  enable  him  to  leverage  opportunities 
by  focusing  on  fewer  unknowns,  clearly  visualizing  current  and  future  end  states,  and 
dictating  the  tempo  within  a  variety  of  environments,  while  being  supported  by  a 
significantly  reduced  staff 

Research  Goals 

Assessment  of  human  functions  in  order  to  improve  human-system  integration  supports 
the  Science  and  Technology  Objective  (STO)  titled  “Methods  and  Measures  of  Commander- 
Centric  Training.”  As  part  of  that  STO  effort,  ARI  is  assessing  the  human  functions  required  for 
command  and  control  of  FCS  forces  equipped  with  advanced  Command,  Control, 
Communications,  Computers,  Intelligence,  Surveillance,  and  Reconnaissance  (C'^ISR)  systems. 


’  Definitions  for  the  acronyms  used  in  this  report  are  presented  in  Appendix  A. 
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The  research  goals  for  this  effort  include  developing  measures  of  individual  and  collective 
command  and  control  performance  to  support  task  allocation,  workload  estimation,  performance 
assessment,  and  training  requirements.  In  support  of  these  goals,  ARI  is  conducting  a  functional 
analysis  of  human  performance  across  the  series  of  PCS  experiments.  This  report  presents  ^ 

interim  findings  on  the  assessment  of  human  performance  based  on  Experiment  3  of  the  PCS  C 
program. 

The  need  for  a  human  functions  assessment  is  underscored  by  the  fact  that  the  PCS 
concept  entails  an  unprecedented  alliance  of  humans  and  machines  at  the  small  unit  level.  This 
interdependence  is  reflected  in  the  PCS  concept  of  a  network-centric  force  composed  of  modular 
manned  and  progressively  autonomous  platforms  with  netted  communication,  sensor,  and  fire 
capabilities.  The  C^  system  prototype  for  the  PCS  C^  program  was  originally  referred  to  as  the 
Commander’s  Support  Environment  (CSE),  but  is  referred  to  as  C^  prototype  in  this  report.  The^ 
CSE  is  a  hardware  and  software  system  located  in  the  command  group’s  C  vehicle.  The  PCS  C 
prototype  includes  workstations  for  each  of  the  command  group  players  Commander,  Battle 
Space  Manager,  Information  Manager,  and  Effects  Manager — ^that  allow  them  to  command  and 
control  their  Unit  Cell  elements  (DARPA,  2001).  The  design  of  the  C^  prototype  must  be  such 
that  it  is  able  to  provide  the  right  information,  at  the  right  place,  and  at  the  right  time,  which  will 
enable  the  command  group  to  fight  emerging  conditions  rather  than  a  predetermined  plan. 

The  functional  analysis  by  ARI  was  designed  to  identify  and  describe  the  command  and 
control  behaviors  of  the  command  group  for  an  PCS  Unit  Cell.  Analysis  developed  detailed 
descriptions  of  critical  command  group  functions,  including  operational  definitions  and 
behavioral  anchors  across  both  the  planning  and  execution  phase  of  selected  runs.  Por 
Experiment  3,  ARI  based  the  human  performance  analysis  on  the  following  measurement 
methods:  objective  measures  of  verbal  interactions;  objective  measures  of  human-computer 
interactions  (HCI)  with  the  PCS  C^  prototype,  including  manual  and  automated  measures  of 
HCI;  and,  subjective  measures  obtained  in  after  action  reviews,  surveys,  and  interviews. 

■  Verbal  Interactions.  Verbal  analysis  of  communications  included  transcription  from  audio 
recordings  of  all  spoken  exchanges  by  members  of  the  command  group  with  one  another, 
with  higher  headquarters,  and  with  subordinate  personnel.  A  taxonomy  of  communications 
was  developed  as  a  structural  framework  for  the  verbal  communications  rating  scheme. 
Verbal  analysis  identified  the  source  and  type  of  communication,  C  function,  subject  matter, 
and  time  duration. 

■  Human-Computer  Interactions.  HCI  analysis  of  player  interactions  included  iterative  review 
of  video  recordings  of  command  group  performance  in  the  C^  vehicle.  A  taxonomy  of  HCI 
tasks  was  developed  as  a  structural  framework  for  the  HCI  rating  scheme. 

■  Automated  Measures.  A  related  ARI  goal  was  to  promote  the  development  of  automated 
measures  of  C^  performance.  ARI  identified  and  validated  a  subset  of  automated  measures 
developed  for  Experiment  3  by  comparing  them  with  corresponding  manual  measures. 

■  Subjective  Measures.  Responses  obtained  from  command  group  players  in  after  action 
reviews,  surveys,  and  interviews  addressed  multiple  research  issues  including:  workload, 
performance  success,  effectiveness  of  the  C^  prototype,  function/task  allocations  among 
humans  and  machines,  teamwork  skills  and  decision  making. 
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As  a  complementary  effort,  ARI-Knox  is  developing  an  in-house  synthetic  C^'lSR  task 
environment  for  analysis  and  experimentation  directed  at  improving  critical  functions, 
human-machine  task  allocation,  system  interfaces,  and  military  job  design. 

Human  Functions  Assessment.  The  term  “functions”  generally  refers  to  groups  of 
related  actions  that  contribute  to  a  larger  action  to  achieve  a  definite  goal  or  purpose.  For 
Experiment  3’s  focus  at  the  Unit  Cell  level,  the  overall  function  of  command  group  actions  was 
to  command  and  control  the  Unit  Cell  and  accomplish  the  assigned  mission.  To  support  the 
assessment  of  human  functions,  a  candidate  set  of  subordinate  command  and  control  functions 
was  adapted  from  the  FCS  C^  experimental  design.  The  set  of  C^  functions  used  by  ARI-Knox 
as  a  fi-amework  for  the  analysis  of  verbal  and  human-computer  interactions  follows: 

■  Plan:  Develop,  assess,  and  modify  a  plan  including  combat  instruction  sets  provided  to 
robotic  elements  in  response  to  changing  events. 

■  See:  Control  and  interpret  input  from  a  heterogeneous  set  of  advanced  sensors  to  mentally 
construct  an  accurate  picture  of  the  battlefield  in  terms  of  Mission,  Enemy,  Terrain,  Troops 
Available  -  Time  and  Civilians  (METT-TC)  factors. 

■  Move:  Control  the  movement  and  activity  of  fnendly  manned  and  unmanned  systems  to 
maintain  desired  movement  rates  and  formations. 

■  Strike:  Distribute  a  variety  of  indirect  and  direct  effects  over  a  set  of  targets. 

■  Battle  Damage  Assessment  (BDA):  Control  and  interpret  input  fi'om  a  heterogeneous  set  of 
advanced  sensors  to  mentally  construct  an  accurate  assessment  of  battle  damage. 

■  Other:  Interactions  that  do  not  fall  in  any  of  the  above  categories. 

To  the  extent  this  is  a  good  set  of  C^  functions,  the  actions  of  the  command  group  during 
the  planning  and  execution  of  experimental  missions,  called  runs,  can  be  usefully  classified  into 
meaningful  C^  behaviors.  This  report’s  assessment  of  human  functions  is  based  on  observable 
behaviors,  particularly  verbal  communications  and  human-computer  interactions,  to  infer  the 
underlying  functions.  Some  degree  of  interpretation  is  necessary  in  classification,  and  any 
behavior  needs  to  be  evaluated  in  context  of  other  behaviors  and  ongoing  mission  events  in  order 
to  infer  function  correctly. 

The  method  approach  taken  to  assess  human  functions  is  to  classify  elements  of  behavior, 
namely  verbal  interactions  and  human-computer  interactions,  into  more  meaningful  command 
and  control  functions.  As  a  result,  the  behaviors  and  workload  demands  associated  with 
command  and  control  of  a  Unit  Cell  can  be  assessed. 

Research  Benefits.  A  detailed  assessment  of  C  functions  and  workload  requirements 
should  support  many  important  FCS  decisions  related  to  manpower,  personnel,  task  allocation, 
materiel,  and  training  requirements.  For  example,  the  HCI  analysis  provides  useful  estimates  on 
the  impact  of  C^  prototype  design  changes  introduced  in  Experiment  3,  such  as  Attack  Guidance 
Matrix  (AGM),  Quick  Fire,  and  Unit  Viewer,  on  command  group  performance  and  Unit  Cell 
effectiveness.  A  related  example,  a  behavioral  taxonomy  that  relates  human-computer 
interactions  to  key  C^  functions  provides  an  empirical  basis  for  the  development  of  automated 
measures  of  command  and  control  performance.  Automated  performance  measures  are  needed 
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for  more  efficient  and  effective  measurement  of,  and  feedback  on,  command  and  control 
performance. 

Method 

The  design  of  Experiment  3  required  that  the  players  plan  and  execute  essentially  the 
same  “See,  Move  and  Strike”  mission  across  1 1  experimental  runs.  The  design  also  allowed 
experimenters  to  vary  run  conditions  as  a  function  of  METT-TC  across  Medium,  High,  and  Too 
High  levels  of  Complexity.  The  execution  portion  of  each  run  lasted  approximately  60-90 
minutes. 

Level  of  Complexity  was  a  key  manipulation  in  Experiment  3,  introduced  to  identify  how 
changes  in  operational  conditions  impact  command  group  player  performance  requirements,  and 
their  resulting  activities  or  human  functions.  It  was  anticipated  that  changing  run  complexity 
level  would  change  workload  and  performance.  More  complete  descriptions  of  the  Unit  Cell’s 
deliberate  attack  mission  and  experimental  manipulations  of  run  complexity  are  provided  in  the 
Experiment  3  Interim  Report  prepared  by  Northrop  Grumman  (available  from  the  PM  PCS  C^). 

■  Medium  Complexity  (Runs  1 ,  3, 7,  and  8)  presented  an  enemy  reconnaissance  force  and 
defending  battalion  (minus)  with  regimental  support. 

■  High  Complexity  (Runs  2,  5,  9,  and  1 1)  presented  a  larger  enemy  reconnaissance  force  and 
defending  battalion  (minus)  with  regimental  support  compared  to  the  Medium  Complexity 
condition.  Runs  at  this  level  of  Complexity  also  entailed  unexpected  “wildcard”  events 
including  threat  force  in  the  South,  school  buses  carrying  “Human  Shields.” 

■  Too  High  Complexity  (Runs  4,  6,  and  10)  presented  a  larger  enemy  reconnaissance  force  and 
defending  battalion  (minus)  with  regimental  support  compared  to  the  High  Complexity 
condition.  Runs  at  this  level  of  Complexity  included  unexpected  “wildcard”  events. 

Objective  Measures.  Prior  PCS  C^  experiments  had  relied  too  heavily  on  subjective 
measures  about  performance  rather  than  direct  measures  of  performance,  particularly  for  human 
performance  assessment.  As  a  result,  for  Experiments  1  and  2  ARI’s  objective  measures  of 
human  performance  were  based  primarily  on  manual  reduction  of  audio  and  video  behaviors 
from  recorded  runs.  The  video  records  were  used  to  conduct  a  relatively  laborious  analysis  of 
the  command  group’s  verbal  interactions  and  human-computer  interactions  with  the  C 
prototype,  as  documented  in  ARI’s  Experiment  2  Interim  Report  (Lickteig,  Sanders,  Durlach,  & 
Carnahan,  2002).  In  Appendix  I  of  that  report,  ARI  submitted  a  “Proposed  Automated  Measures 
List  for  Assessing  Human  Performance  and  Workload.”  As  a  result,  the  PCS  C^  program 
developed  and  implemented  a  subset  of  the  automated  measures  proposed  for  Experiment  3. 

The  ARI  helped  the  PCS  C^  program  validate  the  automated  measures  developed  for 
Experiment  3  by  comparing  them  with  a  corresponding  set  of  manual  measures  based  on  ARI’s 
video  analysis  of  selected  runs.  Results  from  that  analysis  are  provided  in  the  Results  section. 

The  development  and  validation  of  automated  measures  is  an  iterative  process  that  requires  the 
collaborative  efforts  of  behavioral,  technical,  and  operational  subject  matter  experts.  Moreover, 
as  C^  features  are  added  and  refined  corresponding  automated  measures  should  be  identified, 
developed  and  validated.  Hence,  the  iterative  cycle  of  identifying,  defining,  developing,  and 
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validating  automated  measures  of  human  performance  should  extend  into  Experiment  4  and 
future  PCS  development  efforts. 

Subjective  Measures.  Realistically,  an  adequate  assessment  of  human  performance 
requires  a  balance  with  direct  measures  o/performance  and  subjective  measures  about 
performance.  Results  reported  are  based  on  the  battery  of  10  different  subjective  measures 
developed  and  administered  by  ARI  during  Experiment  3. 

Human  Functions  Assessment.  The  Human  Functions  Assessment  discussed  here  in  the 
Method  section  and  in  the  following  Results  section  are  organized  in  the  following  manner: 

■  Verbal  Interactions  (Runs  10  and  1 1). 

■  Human-Computer  Interactions  (Run  10). 

■  Automated  Measures  Development  and  Validation  (Run  1 0). 

■  Subj  ecti ve  Measures  (Runs  1-11). 

Verbal  Interactions 

Methods  for  collection  and  analysis  of  verbal  behavior  from  Experiment  3  are  described 
in  this  section.  As  overview,  the  methods  for  collection  were  similar  to  those  used  for 
Experiment  2.  Methods  for  analysis,  however,  included  method  refinements  designed  to  provide 
a  more  comprehensive  and  meaningful  assessment  of  the  verbal  communications  used  to 
command  and  control  the  Unit  Cell.  The  two  primary  verbal  method  refinements  for  Experiment 
3  are  referred  to  as  Valence  and  Command  Considerations,  described  in  more  detail  later  in  this 
section.  Valence  ratings  were  assigned  to  verbal  behaviors  to  distinguish  communications 
conveying  positive  versus  negative  status  on  accomplishing  basic  C^  functions  and  tasks. 
Command  Consideration  classifications  were  made  on  verbal  behaviors  to  better  identify  how 
participant  communications  related  to  key  cognitive  requirements  considered  fundamental  to  the 
art  of  battle  command. 

Collection  methods  were  based  on  a  form  of  Verbal  Protocol  Analysis  (Ericsson  & 
Simon,  1984,  1993)  used  in  prior  PCS  C^  experiments.  In  essence,  participants  were  encouraged 
to  “think  out  loud”  during  the  planning  and  execution  phases  of  each  run.  Audio/video 
recordings  from  selected  runs  were  used  to  collect  the  primary  participants  verbal 
communications  with:  one  another  in  the  C^  vehicle  (Within  Cell);  higher  echelon,  the  Blue 
Team  (Black/Blue);  subordinate  driver  and  gunner  in  the  C^  vehicle  (Black/  Subordinate);  and, 
supporting  personnel  including  technical  assistants  (Other).  Communications  were  reviewed  and 
transcribed  in  conjunction  with  a  quadraplex  video  from  over-the-shoulder  cameras  behind  each 
primary  participant.  This  quad  video  image  of  the  four  primary  participants  in  the  command 
group  greatly  assisted  verbal  transcription  and  coding,  such  as  identifying  the  Source  of  each 
communication. 

The  analysis  of  verbal  communications  was  limited  to  the  final  two  runs  of  Experiment  3, 
Run  10  (Too  High  Complexity)  and  Rim  1 1  (High  Complexity).  Verbal  analysis  of  all  1 1  runs 
was  not  attempted  due  to  the  extensive  time  and  labor  required  to  transcribe,  chunk,  and  rate  all 
communications  during  planning  and  execution  phases.  Rationales  for  selecting  the  final  runs 
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were  to  ensure  a  more  stable  experimental  environment,  and  more  proficient  performance  by  the 
command  group  due  to  the  deliberate  practice  design  of  the  experiment. 

In  coding  of  verbal  data,  there  is  often  a  trade-off  between  extracting  the  cognitive 
processes  of  the  speakers  in  a  meaningful  manner  versus  achieving  inter-coder  reliability. 
Chunking  and  classifying  communications  into  meaningful  categories  often  requires  a  degree  of 
interpretation  and  context  that  tends  to  broaden  the  scope  of  disagreement  between  coders.  If 
different  coders  fail  to  agree,  then  one  must  question  the  reliability  of  the  results.  One  way  to 
increase  inter-coder  agreement  is  to  consider  smaller  chunks  or  samples  of  verbal  data  tin  order 
to  narrow  the  range  of  interpretation.  The  ARI  used  this  approach,  smaller  verbal  chunks,  during 
Experiment  2  and  achieved  average  inter-coder  agreement  across  coding  categories  of  93.2%. 
Based  on  that  success,  smaller  verbal  chunks  were  also  used  to  analyze  the  verbal 
communications  for  Experiment  3. 

Initially,  all  verbal  communications  from  Runs  10  and  1 1  were  converted  to  written 
transcripts  that  were  appended  with  data  on  source  and  time  of  communication.  These 
transcripts  were  subsequently  "chunked.”  That  is,  the  flow  of  communication  was  blocked  into 
units  amenable  for  subsequent  coding.  This  chunking  of  the  transcript  required  a  researcher  to 
evaluate  the  transcript  and  then  to  group  a  cluster  of  dialog  together  that  appeared  to  be  unitary 
and  consistent.  The  goal  of  chunking  was  to  create  coherent  blocks  of  dialog  that  were  specific 
enough  in  they  did  not  fall  under  multiple  rating  categories.  Initial  chunking  was  done  on  the 
basis  of  the  Type  category  of  the  coding  scheme  (see  next  section.  Verbal  Coding  Categories). 
Verbal  chunks  were  further  divided  to  ensure  xmique  codes  for  all  categories.  Finally,  the 
researcher  who  led  the  verbal  analysis  effort  for  Experiment  2,  assigned  codes  for  all  categories 
included  in  the  verbal  coding  scheme  for  Experiment  3. 

Verbal  Coding  Categories.  Verbal  behaviors  from  Experiment  3  were  analyzed  using  a 
revised  version  of  the  verbal  coding  scheme  developed  and  used  for  Experiment  2  (Lickteig,  et 
al.,  2002).  A  brief  description  of  the  verbal  coding  scheme  used  for  Experiment  3  is  provided  in 
this  section.  Appendix  B  provides  more  detailed  documentation  on  the  verbal  coding  scheme 
used  for  Experiment  3. 

Verbal  conunxmications  were  analyzed  for  the  categories  of  Source,  Function,  Type,  and 
Factor  used  for  Experiment  2.  The  categories  of  Valence  and  Command  Considerations  were 
also  developed  and  used  for  Experiment  3,  as  noted.  The  coding  schemes  for  Source,  Function, 
Type,  and  Factor  were  as  follows: 

■  Source  coded  the  verbal  behavior  for  who  was  speaking  to  whom. 

■  Function  coded  the  verbal  behavior  for  C^  Functions:  See,  Plan,  Move,  Strike,  Battle 
Damage  Assessment  (BDA). 

■  Type  coded  the  purpose  of  verbal  behavior:  Share,  Action,  Direction,  Ask,  Process,  Decide. 

■  Factor  coded  the  verbal  behavior  for  relation  to  Mission,  Enemy,  Terrain,  Troops,  Time,  and 
Civilians  (METT-TC)  Factors.  As  for  Experiment  2,  each  METT-TC  Factor  was  subdivided 
into  more  specific  sub-factors  (e.g..  Enemy  Location,  Identification  and  Disposition)  for  a 
total  of  25  sub-factor  categories  (see  Appendix  B  for  more  detail). 
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Notably,  the  System  category  was  dropped  from  the  Experiment  3  coding  scheme.  For 
Experiment  2,  the  System  category  was  introduced  to  more  precisely  assess  how  FCS  related 
assets  (e.g.,  Roboscout,  Micro-Unmanned  Aerial  Vehicle  (UAV),  Shadow  UAV)  were  utilized  or 
discussed  by  the  command  group  players.  System  coded  the  specific  Unit  Cell  asset  that  was  the 
primary  focus  of  each  verbalization  chunk.  The  System  category  was  not  used  for  Experiment  3, 
however,  as  it  provided  insufficient  additional  information. 

Valence.  A  perceived  shortcoming  in  the  Experiment  2  verbal  scheme  was  a  failure  to 
account  for  the  evaluative  information  conveyed  by  the  verbal  communications  of  the 
participants.  What  communications  conveyed  positive  versus  negative  information?  Particularly, 
what  did  the  communications  convey  on  the  status  of  accomplishing  functions  and  tasks? 

Evaluative  meaning  is  conveyed,  or  sought,  in  nearly  all  types  of  commimication.  For  command 
and  control  communications,  the  evaluative  information  shared  in  communication  should 
directly  support  the  command  group’s  decision  making  process. 

For  example,  the  statements  “I  can’t  get  it  to  move”  and  “He  is  moving  on  plan  at  30 
miles  per  hour”  would  receive  identical  codes  based  on  the  Experiment  2  verbal  coding  scheme 
without  Valence  consideration.  While  these  statements  are  “about”  the  same  Move  function, 
they  convey  very  different  information  on  the  accomplishment  of  that  function.  Thus,  Valence 
was  introduced  to  distinguish  communications  conveying  positive  versus  negative  status  on 
accomplishing  basic  functions  and  tasks. 

For  Experiment  3,  each  verbal  chunk  was  scored  as  negative  (-1),  neutral/  inconclusive 
(0),  or  positive  (1)  with  respect  to  its  coded  function.  For  example,  “I  can’t  get  it  to  move”  was 
assigned  a  negative  value  (-1);  “Is  it  moving?”(without  a  verbal  response  to  the  question)  was 
assigned  a  neutral  value  (0);  and  “He  is  moving  on  plan  at  30  miles  per  hour”  was  assigned  a 
positive  value  (1).  The  valence  codes  do  not  address  the  tactical  goodness  or  appropriateness  of 
the  subject  function  or  task,  only  whether  it  was  being  performed  as  anticipated,  or  not  being 
performed  as  anticipated.  Valence  codes  for  the  BDA  function,  however,  require  further 
explanation.  BDA  verbalizations  were  scored  as  negative  (-1),  if  (a)  no  BDA  images  were 
available  or  useful,  or  (b)  images  were  available,  but  the  images  indicated  that  the  enemy  asset 
still  posed  a  threat.  Table  1  provides  examples  of  the  Valence  codes  assigned  to  verbal  chunks 
from  Run  10  for  each  function. 
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Table  1 


Examples  of  verbal  chunks  from  Run  10  by  Function  with  assigned  Valence  values  of  positive 
(1),  neutral/inconclusive  (0),  and  negative  (-1) 


Valence 

Function 

Examples  of  Verbal  Chunks 

Plan 

1 

I  was  just  thinking  about  the  birds  being  too  far  up  there,  up  North 

Bring  them  South  then. 

I  will,  I  don't  control  anything,  I've  got  to  ask  team  to  bring  them  further 
South.  And  I  can  do  that,  sir,  if  you  don't  mind. 

No  go  right  ahead,  keep  them  South,  that's  fine  with  me. 

0 

I  got  an  idea,  do  you  want  to  try  something  new? 

No. 

-1 

We  have  unconfirmed  as  of  yet  BDA  on  a  tank  in  the  South  and  a  couple  of 
tanks  in  center  sector  but  we  don't  have  enough  intelligence  yet  to  give 
us  a  good  read  of  the  battlefield  other  than  the  fact  that  he  tried  to  move 
forward  in  the  center,  and  I  will  keep  you  informed. 

See 

1 

There’s  an  unknown  radio  hit. 

0 

They  haven’t  fired  any  artillery  yet,  have  they?  (no  response) 

-1 

Dang,  there’s  nothing  in  my  images  here. 

Move 

1 

So  we  need  to  get  those  2  micros  back  down  there. 

They’re  coming  down. 

0 

Where  is  the  Synthetic  Aperture  Radar  (SAR)  bird?  (no  response) 

-1 

That  one’s  stuck  there,  number  2  is  just  not.. .  micro-UAV  2  is  not 
responding. 

Strike 

1 

Did  you?  Did  you  fire  4? 

Yeah,  I  just  fired  4. 

0 

Well,  the  question  is,  do  we  reengage?  (no  response) 

-1 

OK.  Interestingly,  you  lost  comms  on  the  Precision  Attack  Missile 
(PAMs)  that  you  sent.  You  see. 

Table  Continues 
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that?  PAM  54  lost  comms.  It  didn’t  attack.  Hold  on  a  second,  the  last  2 
PAMs  that  you  sent  lost  comms  and  did  not  go  to  the  target.  You  want 
me  to  show  you?  The  one  on  the  Darya  did  that  too.  Neither  one  hit 
anything.  They  both  lost  comms. 

BDA 

1 

Here  is  a  better  image.  It  looks  like  it  might  be  perhaps  a  fire  power  kill, 
maybe  a  fire  power,  mobility  kill. 

0 

PAM  16,  where  did  that  hit?  (no  response) 

-1 

Is  it  broke?  Did  we  kill  it?’ 

I  don't  know,  it  doesn't  look  like  it's  broke  from  this  image  right  here,  it’s 
hard  to  tell. 

Other 

1 

What’s  the  red  dot  mean? 

It  means  that’s  where  it  detected  something  and  takes  a  picture,  or  that's  the 
place  where  the  Garm  was  templated. 

0 

Blue  6,  Black  6  (no  response). 

-1 

I’ve  got  a  right  screen  frozen. 

Command  Considerations.  The  second  method  refinement  to  verbal  analysis  for 
Experiment  3  was  the  addition  of  a  coding  category  called  Command  Considerations.  Another 
perceived  shortcoming  in  the  verbal  analyses  for  Experiment  1  and  2  was  the  need  for  a  more 
explicit  relation  between  thought  and  action,  particularly  action  in  the  form  of  verbal  interaction. 
How  do  the  verbal  behaviors  of  the  participants  relate  to  the  cognitive  processes  required  for 
battle  command?  Notably,  Army  research  on  battle  command  has  begun  to  identify  a  basic  set 
of  cognitive  themes  or  mental  scripts  that  commanders  use  to  organize  and  visualize  the 
battlefield  (Lussier,  Shadrick,  «fe  Prevou,  2003). 

For  Experiment  3,  an  exploratory  set  of  Command  Considerations  was  developed  to 
relate  participant  communications  with  key  cognitive  patterns  important  to  battle  command.  The 
nine  (9)  topics  used  to  assess  Command  Considerations  are  listed  in  Table  2.  Coding  of  the 
verbal  data  for  Command  Considerations  did  not  treat  the  topics  as  exclusive.  That  is,  the  same 
verbal  chunk  could  be  could  be  coded  under  multiple  topics  or  considerations. 
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Table  2 


Command  Considerations  for  Analyzing  Command  Group  Verbalizations 


Consideration 

Description  of  Consideration 

Plan 

Inform 

See 

Coordinate 

Assets 

Situation  Awareness 
TerrainATime 

Enemy 

Mission 

Execution  is  self-initiated  and  preceded  by  plan  coordination/refinement. 
Make  information  requirements  known. 

Battlefield  visualizations  that  are  dynamic/predictive/proactive. 

Create  synergistic  effects  with  multiple  assets/teamwork. 

Use  all  assets  available. 

Continual  situation  assessment,  dynamic/contingency  planning. 

Consider  effects  of  terrain/time. 

Model  a  thinking  enemy. 

Keep  sight  of  the  big  picture  and  mission  intent. 

Planning.  At  ARI’s  request,  the  planning  phase  for  each  run  during  Experiment  3  was 
recorded,  in  addition  to  the  execution  phase.  Verbal  interactions  during  planning  were  analyzed 
for  Runs  1 0  and  1 1 ,  in  the  manner  previously  described  for  the  execution  phase.  Fidelity  of  the 
verbal  recordings  was  lower  than  during  execution,  however,  due  to  participants  neglecting  to 
use  their  headsets  continuously  during  planning  and  preparation.  Notably,  it  was  observed  that 
some  planning  discussions  took  place  outside,  as  well  as  within,  the  C^  vehicle.  So  not  all  verbal 
interactions  related  to  planning  were  recorded  or  analyzed.  Nevertheless,  results  on  plaiming  are 
provided  in  the  results  section. 

Human-Computer  Interactions 

2 

Methods  for  analyzing  human-computer  interactions  (HCI),  command  group-C 
prototype  interactions,  were  reviewing,  coding  and  quantifying  data  from  video  recordings  of 
planning  and  execution  phases  from  Run  10  (Too  High  Complexity).  A  quality  versus  quantity 
analytic  strategy  focused  on  a  relatively  comprehensive  analysis  of  HCI  performance  during  a 
single  run  versus  a  coarser  analysis  of  multiple  runs.  Separate  video  recordings  were  captured  at 
each  of  the  two  screens  at  eaeh  of  the  four  command  group  player  workstations  (eight  recordings 
total).  These  video  recordings  were  reviewed  and  HCI  records  of  the  behavioral  tasks  performed 
were  developed.  A  Head  Up  Display  (HUD)  flat  panel  was  mounted  to  the  ceiling  in  the  forward 
area  of  the  C^  Vehicle,  which  allowed  the  Commander  and  Battle  Managers  to  view  a  common 
screen.  The  players  could  toggle  any  of  their  eight  screens  or  the  driver/gunner’s  display  to  the 
HUD  at  any  time.  However,  no  video  record  was  made  of  the  information  presented  on  the 
HUD. 


The  HCI  Rating  Scheme  Development.  Based  on  ARI’s  understanding  of  the  new 
features  developed  in  the  C^  prototype  for  Experiment  3,  and  a  review  of  all  Run  10  video 
recordings,  the  HCI  rating  scheme  developed  for  Experiment  2  was  revised  and  expanded  (see 
Figure  1).  As  illustrated  in  Figure  1,  the  evaluation  framework  for  HCI  remained  structured  to 
basic  C^  Functions  (Plan,  See,  Move,  Strike)  and  Sub-Functions.  However,  the  number  of  HCI 
tasks  identified  increased  from  53  to  83  to  reflect  the  new  features  added  to  the  C^  prototype  for 
Experiment  3,  and  ARI’s  focus  on  documenting  the  HCI  tasks  performed  during  planning  as  well 
as  execution. 
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An  excerpt  of  the  HCI  rating  scheme  is  provided  in  Table  3  for  method  clarification.  A 
three-digit  code  based  on  the  HCI  rating  scheme  was  applied  to  each  interaction.  The  first  digit 
designates  the  Function  (Plan,  See,  Move,  Strike).  The  second  digit  designates  the  Sub- 
Function  (17  total).  The  third  digit  designates  the  83  supporting  HCI  tasks.  For  example,  the 
three  digit  code  “113”  designates:  the  Plan  C^  Function,  the  Create  Mission/Course  of  Action 
(COA)  C^  Sub-Function  and  the  Rehearse  Plan  task  or  interaction. 

Table  3 

Excerpt  of  HCI  Rating  Scheme 


100  PLAN  (Describe  Tactical  Situation,  Concerns,  and  Future 
Activities,  Request  Information) 

110.  Create/Update  a  Mission  and  COA 

111.  Create  Overlay  Graphics  and  Map  Annotations 

1 12.  Place  platforms  on  the  map  (friendly  and  threat  template) 

113.  Rehearse  Plan 

114.  Execute  Plan 

115.  Point  on  Map  Using  Cursor/Indicate  an  Area 

1 16.  Move  Icons  (vehicle  or  GCM)  on  Map  Using  Drag/Drop 

1 1 7.  Modify  overlay  graphics 

120.  Alerts 

121,  Create  Alerts 


This  HCI  taxonomy  or  evaluation  framework  facilitates  comparisons  across  FCS  C^ 
experiments,  and  should  support  FCS  task  identification  in  future  efforts.  Iterative  revision  of 
the  taxonomy  and  rating  categories  occurred  during  the  analysis  of  video  recordings.  Two  ARI- 
Knox  researchers  rated  each  of  the  video  recordings  from  Experiment  3  Run  10  execution  and 
planning  phases.  An  example  of  HCI  scoring  and  annotations  for  the  Battlespace  Manager’s 
right  screen  during  Run  10  execution  is  provided  in  Table  4  to  clarify  the  analytic  approach. 
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The  HCI  task  performance  criteria  included  task  frequency,  duration,  and  errors. 
Transcripts  described  each  HCI  task  performed,  recorded  Start  and  Stop  times  where  appropriate, 
and  annotated  any  performance  errors  observed.  For  HCI  tasks  that  routinely  required  5  seconds 
or  less  to  perform,  such  as  using  Zoom  and  Scroll  map  tools,  times  were  not  recorded.  However 
special  attention  was  paid  to  record  Start  and  Stop  times  for  any  instances  where  tasks  took 
longer  than  5  seconds  to  perform.  The  HCI  task  records  and  ratings  were  compared  to  generate 
an  estimate  of  inter-rater  reliability.  The  independent  ratings  yielded  an  index  of  agreement 
between  raters  of  96%.  Results  from  the  analysis  are  documented  in  the  Results  section. 

Table  4 

Example  of  Task  Scoring  Using  HCI  Rating  Scheme 


Start 

Time 

Stop 

Time 

Code 

Description 

0  00  00 

Map  80%,  Execution  window  20% 

0  00  38 

325 

Expand  Execution  window  to  50%  screen 

0  01  04 

344 

Open  State  View  window,  deselect  templated  targets 

KfilMM 

0  01  41 

325 

Expand  map  to  100%,  takes  a  long  time  to  change 

iHBI 

332 

Target  Query  Automatic  Target  Detection(ATD)/Track 
(TK)  A-22 

332 

Target  Query  ATD/TK  A-22 

0  04  31 

0  04  40 

332 

Target  Query  ATD/TK  A-22 

0  05  10 

311 

Zoom  Map  Out 

0  08  54 

Fire  Query  Intemetted  Unattended  Ground  Service 
(lUGS)  1 

0  08  59 

333 

Fire  Query  lUGS  3 

0  09  10 

Enemy  Intel,  delete,  confirm 

0  09  20 

HU 

Select  vehicle  from  menu,  ATDl,  remove  from  map 

011  44 

332 

Chaser  Query,  PAM  16,  mode  =  detonated 

0  11  51 

334 

Target/Chaser  Query,  unknown  7 

0  11  54 

332 

Chaser  Query,  Chaser  13 

442 

PAM  Query,  PAM  21  Mode  attacking 

351 

Select  target  from  menu.  Chaser  6,  Remove  From  Map 

*  Information  Manager’s  Right  Screen,  Run  10  Execution  Phase 


HCI  Data  Reduction.  For  Experiment  3,  high-resolution  video  recordings  of  each  of  the 
eight  command  group  screens  for  Run  10  planning  and  execution  were  transferred  to  computer 
hard  drive  media  for  review  by  ARI-Fort  Knox  researchers.  The  Run  10  Execution  phase  was  83 
minutes  in  length,  so  that  the  eight  video  recordings  represented  a  total  of  approximately  12 
hours  of  video  data.  The  Run  10  Planning  phase  lasted  56  minutes,  with  the  eight  video 
recordings  providing  approximately  eight  hours  of  data.  The  review  of  each  hour  of  screen  video 
recording  required  approximately  six  hours  of  researeher  time  to  create  the  HCI  task  record 
(annotated  with  task  times)  and  to  rate  the  transcript  using  the  HCI  rating  scheme.  After  rating 
scheme  refinement  and  expansion  to  accommodate  new  C^  prototype  features  and  tasks  from 
Experiment  3,  approximately  16  days  of  researcher  time  was  required  to  create  all  HCI  task 
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records  for  Run  10.  Records  for  the  eight  command  group  screens  ranged  in  length  from  39  to 
268  tasks  for  the  Execution  phase  of  Run  10,  and  from  1  to  164  tasks  for  the  Planning  phase. 

Automated  Measures  Development  and  Validation 

Automated  performance  measures  are  required  to  support  training,  evaluation  and  C 
system  design  (Unit  of  Action  Maneuver  Battle  Laboratory,  2002).  This  requirement  applies  to 
the  PCS  prototype,  and  to  all  future  systems.  Manual  video  data  reduction  of  command 
and  control  performance  can  only  examine  a  fraction  of  the  data  potentially  available  from  each 
PCS  experiment,  or  any  future  PCS  training,  testing  and  evaluation  effort.  Por  Experiment  3 
the  human-computer  interaction  (HCI)  data  for  only  one  (Run  10)  of  the  eleven  runs  was 
examined.  Manual  data  reduction  required  approximately  sixteen  (16)  days,  as  previously  noted, 
to  identify  and  tabulate  the  1,043  human-computer  interactions  that  occurred  during  Run  10 
execution,  and  the  499  human-computer  interactions  that  occurred  during  Run  10  planning. 

In  contrast,  automated  logs  of  human-computer  interactions  provide  efficient  and 
effective  measures  of  command  and  control  performance.  The  efficiency  of  automated  measures 
equates  to  quick  and  inexpensive.  It  includes  the  ability  to  adjust  the  range  and  selection  of  data 
to  include  the  performance  of  any  or  all  users  at  any  or  all  times  during  an  operational 
exercise.  The  effectiveness  of  automated  measures  equates  to  increased  scope  and  precision  in 
the  collection  of  performance  data.  It  includes  more  meaningful  measures  by  automatically 
correlating  performance  with  the  battlefield  situation  in  which  it  occurred. 

Por  Experiment  3,  ARI  requested  a  select  set  of  key  automated  measures  be  developed  to 
support  human  performance  assessment  for  human-computer  interaction,  particularly  command 
group  and  higher  interactions  with  their  prototypes.  Table  5  lists  and  describes  the  automated 
measures  requested.  Members  of  the  Technical  Team  created  software  that  automatically 
logged  3  of  the  23  automated  measures  requested,  as  indicated  in  Table  5.  A  supporting 
contractor  team  captured  the  logs  for  these  three  measures  from  the  prototypes  of  each  player 
and  higher  headquarters  (Blue  Team)  workstation,  at  the  end  of  each  run  execution  phase. 
Notably,  human-computer  interactions  during  planning  were  not  captured.  The  same  contractor 
team  then  reduced  the  logged  data  in  support  of  their  own  documentation  requirements,  and 
provided  ARI  a  set  of  parsed  metric  files  that  could  be  viewed  and  modified  in  commercially 
available  spreadsheet  applications. 

The  focus  of  ART  s  effort  was  to  help  the  PCS  program  develop,  refine  and  validate  a 

relatively  comprehensive  set  of  automated  measures  to  support  training,  evaluation  and  C 
system  design.  For  Experiment  3,  this  effort  was  limited  to  the  3  automated  measures  actually 
developed  by  the  Technical  Team.  The  method  approach  for  validation  of  these  3  measures  was 
to  compare  the  log  data  obtained  with  corresponding  manual  measures  from  ARI’s  HCI  analysis. 
Although  data  logs  of  automated  measures  were  obtained  for  all  1 1  runs  of  Experiment  3, 
comparison  was  limited  Run  10  as  that  was  the  only  run  manually  reduced  for  HCI  analysis. 

The  logged  data  provided  to  ARI  consisted  of  5  possible  files  per  command  group  player, 
totaling  as  many  as  20  parsed  files  per  record  run.  The  total  number  of  files  was  dependent  on 
whether  or  not  the  player  performed  the  necessary  tasks/keystrokes  to  activate  a  log  tabulation  of 
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the  event  by  the  software  capturing  the  automated  measures.  The  logged  data  provided  to  ARI 
also  included  data  from  higher  headquarters  (Blue  Team),  but  comparison  with  manual  data  was 
not  possible  as  video  recordings  of  Blue  Team  workstation  were  not  made.  The  five  log  files  for 
each  command  group  player  were  as  follows: 

■  The  alerts  acknowledged  during  the  record  run. 

■  Images  requested. 

■  Image  requester  tag  (how  the  image  was  requested  from  the  system). 

■  Image  not  available  tag. 

■  Create  route  tag  for  ground  and  air  units. 

Results  on  ARI’s  effort  to  help  the  PCS  program  develop  and  validate  automated  measures 
are  provided  in  the  Results  section. 

Table  5 

ARI’s  Requested  Automated  Measures  List  for  Experiment  3 


Measurement  Category 

Automated  Measure _ _ _ 

See 

Number  of  pictures/images  available. 

Number  of  pictures/images  with  actual  enemy  image  of  those  available. 

*  Number  of  pictures/images  opened. 

Amount  of  time  manipulating  pictures  (zoom,  contrast,  pan)  to  improve  image. 

Number  of  times  same  picture  opened  by  same  individual. 

Number  of  times  same  picture  opened  by  different  individuals. 

Alerts 

Number  and  type  of  alerts  set  by  duty  position. 

Number  and  type  of  alerts  triggered/activated  by  duty  position. 

Time  to  respond  to  alerts  by  turning  it  off. 

Number  of  times  robotic  vehicles  auto  halt  (red  line  imder  the  vehicle). 

Time  to  respond  to  auto  halt  to  re-tasking  the  entity. 

Number  of  fi'atricide  warnings  (identify  shooter/target  pair). _ 

Move  Assets 

*  Number  of  times  and  task  duration  for  Create  a  Route  (ground  platform). 

*  Number  of  times  and  task  duration  for  Create  a  Route  (air  platform). 

Strike 

Number  of  times  weapon  fired  and  task  duration  (Network  Fire  [Netfire],  Line  of  Sight  [LOS], 
Infantry  Fighting  Vehicle  [IFV]). 

Number  of  times  “Reassign”  menu  option  selected  for  Loitering  Attack  Missile/Munitions 
(LAM). 

Number  of  times  target  Type  changed  by  selecting  “Apply”  button  in  Recon  Window. 

Number  of  times  Target  Status  modified  by  selecting  “Suspected,”  “Targeted,”  “Dead,”  etc. 
Number  of  times  sensors  tasked  to  recon  targets  by  selecting  “OK”  fi-om  Recon  Target  menu. 

Table  Continues 
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Assess  Icon  and  Map  Information 

Number  of  times  cursor  moved  over  a  platform  to  bring  up  information  window. 
Number  of  times  “Templated”  targets  toggled  off  in  status  toolbar. 

Number  of  times  Zoom  Map  options  selected  (box  and/or  arrows  menu  option). 
Number  of  times  scroll  arrows  used  for  map  manipulation. _ 


*Designates  the  three  (3)  automated  measures  developed  for  Experiment  3. 

Subjective  Measures 

A  battery  of  10  different  subjective  measures  were  developed  and  administered  by  ARI 
during  Experiment  3.  Three  of  these  questioimaires  were  used  during  earlier  PCS 
experiments.  The  seven  (7)  additional  questionnaires  are  new  instruments  designed  to  provide 
more  precise  information  on:  workload  (2  questionnaires),  the  new  prototype  features 
introduced  in  Experiment  3  (2  questionnaires),  higher-order  teamwork  and  decision  making 
skills  (2  questionnaires)  and  finally,  repeated  self-assessments  of  players’  technical  and  tactical 
proficiency  (1  questionnaire).  These  measures  are  described  below  and  documented  in 
Appendix  H. 

In-Place  After  Action  Review  (AAR).  Immediately  after  the  completion  of  each  run,  a 
five  to  ten  minute  structured  interview  or  “In-Place  AAR”  occurred.  This  interview  was 
conducted  with  player  personnel  still  at  their  designated  workstations  in  the  mock-up  Vehicle. 
This  setting  allowed  players  to  review  and  refer  to  their  tactical  displays  as  they  provided  a 
summary  of  their  performance  during  the  run.  This  setting  also  supported  video  and  audio 
recording  of  all  player  comments  and  references  to  their  tactical  displays.  At  the  completion  of 
the  run,  an  ARI  researcher  entered  the  Vehicle  and  read  a  scripted  question  to  each  player  that 
asked  for  a  brief  recap  of  “what  went  right  and  what  went  wrong”  during  the  run  relative  to  their 
duty  position.  To  collect  input  from  the  three  Battle  Staff  Managers  prior  to  the  Unit  Cell 
Commander,  the  pre-specified  order  for  player  interviews  was:  Battlespace  Manager, 
Information  Manager,  Effects  Manager,  and  finally  the  Commander.  These  recorded  interviews 
were  also  used  to  cue  analysts  to  watch  for  specific  tasks  and  operational  events  during  HCI 
analysis  of  the  recorded  runs. 

Workload  and  Performance.  Immediately  after  the  In-Place  AAR,  players  exited  the  C^ 
Vehicle  and  completed  a  brief  survey  on  Workload  and  Performance  Success.  Players  rated  their 
perceived  workload  across  five  dimensions:  Mental,  Physical,  Temporal,  Effort,  and  Frustration 
(1  =  Low  to  100  =  High).  The  workload  questions  and  dimensions  were  adapted  from  the 
•  relatively  standard  Task  Load  Index  (TLX)  developed  by  National  Aeronautics  and  Space 

Administration  (NASA) — ^Ames  Research  Center  (1986).  Performance  success  was  rated  on  this 
same  questionnaire  (1  =  Failure  to  100  =  Perfect). 

C'  Prototype  Support  of  PCS  Functions  and  METT-TC  Factors.  After  the  final  run 

(11),  the  players  provided  estimates  of  the  C^  prototype’s  effectiveness  for  C^  functions  (Plan, 
Move,  See,  and  Shoot)  and  METT-TC  factors  (Mission,  Enemy,  Troops,  Terrain,  Time,  and 
Civilians).  Players  were  asked  to  rate  “How  effective  was  the  C^  prototype  in  support  of  C2 
functions  and  METT-TC  factors?”  using  a  five-point  rating  scale  (1=  Very  Ineffective  to  5  = 
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Very  Effective),  with  an  additional  “Not  Applicable”  response  option.  The  questionnaire 
included  definitions  for  each  function  and  factor,  and  provided  space  for  comments. 

Skill  Proficiency.  The  increased  complexity  expected  in  PCS  combat  systems  and  future 
combat  conditions  will  place  a  premium  on  technical  and  tactical  proficiency.  Given  these 
concerns,  a  survey  instrument  was  prepared  and  administered  repeatedly  to  gather  self-reports  of 
technical  and  tactical  skills  for  key  individual  and  collective  tasks.  The  same  questionnaire  was 
administered  after  training  (Post-Train)  and  after  Runs  3,  5,  and  9  to  obtain  estimates  of  initial 
proficiency,  and  proficiency  as  a  function  of  practice.  A  “Comments”  section  in  the  survey 
solicited  additional  feedbaek  from  participants  on  skill  proficiency. 

Players  were  asked  to  rate  skill  proficiency  on  a  9-point  scale  (1  =  Extremely  Low 
Proficiency  to  9  =  Extremely  High  Proficiency).  Individual  skills  questions  addressed  teehnical 
proficiency  in  using  the  prototype,  and  tactical  proficiency  at  their  assigned  duty  position. 
Collective  technical  skill  questions  asked  players  to  rate  the  team’s  profieiency  in  using  the  (f 
prototype  to  perform  collective  tasks,  and  to  direct  the  actions  of  robotic  assets.  Collective 
tactical  skill  questions  asked  players  to  rate  the  team’s  proficiency  in  communicating 
(gathering/sharing  information)  and  making  key  decisions. 

Workload  on  New  Cf  Prototype  Features.  Automation  does  not  always  reduce  workload. 
To  more  precisely  examine  workload,  a  new  questionnaire  was  developed  on  selected  new  C^ 
prototype  features  added  for  Experiment  3.  The  questionnaire  addressed  eight  new  features,  and 
asked  participants  to  rate  their  workload  related  to  each  feature  on  a  5-point  scale  (1  =  Increased 
Greatly  to  5  =  Decreased  Greatly).  The  survey  was  administered  after  Run  2  to  allow 
participants  to  gain  some  familiarity  with  the  new  features,  and  after  Run  9  to  identify  whether 
perceived  workload  decreased  with  practice  using  the  new  features. 

New  Prototype  Features  Effectiveness.  The  “New  CSE  Features  Effectiveness” 
questionnaire  was  developed  to  more  precisely  assess  the  value  of  the  new  Cf  prototype  features 
inserted  for  Experiment  3.  This  questionnaire  asked  the  participants  to  rate  the  effectiveness  of 
13  new  prototype  features  on  a  5-point  scale  (1  =  Very  Ineffective  to  5  =  Very  Effeetive).  The 
questionnaire  was  administered  at  the  conclusion  of  Run  5  after  the  players  gained  some 
familiarity  with  the  automated  features,  and  again  after  Rim  10  to  determine  if  their  views  of  the 
effectiveness  of  the  new  features  changed. 

Prototype  Support  of  C^.  To  complement  ARI’s  assessment  of  human-computer 
interactions  (HCI),  a  questionnaire  was  developed  and  administered  to  assess  how  effectively  the 
Experiment  3  prototype  supported  a  set  of  HCI  tasks  identified  after  Experiment  2.  The  set  of 
12  HCI  tasks  examined  are  identified  in  Figure  8.  Players  were  asked  to  rate  the  effectiveness 
(1  =  Very  Ineffective  to  5  =  Very  Effective)  of  the  C^  prototype  in  support  of  each  task.  The 
questionnaire  was  administered  after  Run  8. 

Teamwork  Skills.  For  more  effective  battle  eommand,  research  must  focus  on  team 
process  measures  that  underlie  good  command  group  performance,  and  not  just  battle  outcomes. 
An  open-ended  questionnaire  was  administered  that  asked  the  players  to  provide  examples  of 
effective  and  ineffective  C^  teamwork  skills.  The  four  teamwork  skills  addressed  by  and  defined 
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in  the  questionnaire  were:  Communication,  Coordination,  Performance  Monitoring  and 
Feedback,  and  Shared  Situational  Awareness.  Due  to  procedural  disruptions,  the  Commander 
and  Information  Manager  completed  the  questionnaire  after  Run  5,  and  the  Battlespace  and 
Effects  Managers  completed  the  questionnaire  after  Run  7. 

Decision  Making.  A  primary  goal  of  the  FCS  &  system  design  is  to  provide  the 
command  group  with  the  information  necessary  to  make  effective  decisions.  While  decision 
making  might  be  inferred  based  on  the  allocation  of  functions  and  assets  by  duty  position, 
inference  is  still  required.  An  open-ended  questionnaire  was  administered  that  asked  the  players 
to  describe  “Important  Decisions”  they  had  made  in  the  preceding  run,  and  to  identify  C^ 
prototype  features  that  supported  the  decisions  made.  Due  to  procedural  disruptions,  the 
Commander  and  Information  Manager  completed  the  questionnaire  after  Run  5,  and  the 
Battlespace  and  Effects  Managers  completed  the  questionnaire  after  Run  7. 

Human  Systems  Integration  Questionnaire.  The  ARI’s  efforts  to  improve  measurement 
methods  included  adaptation  of  an  instrument  used  by  the  Army  Research  Laboratory  (ARL)  to 
assess  Apache  Helicopter  64D  (AH-64D)  Longbow  Crew  Stations  (Durbin,  2002).  The 
Longbow  may  be  a  key  component  in  future  FCS  organizations.  This  Human  Systems 
Integration  Questionnaire  addressed  two  basic  areas:  Task  Workload  and  C^  prototype  usability. 
The  Task  Workload  dimension  was  designed  to  complement  ARI’s  Experiment  2  approach  for 
assessing  HCI.  Players  were  asked  to  rate  their  workload  on  a  set  of  HCI  tasks  based  on  a  10- 
point  scale  (1  =  “Workload  Insignificant”  to  10  =  “Task  Abandoned”).  Players  were  also  asked 
to  identify  additional  tasks  and  recommend  improvements  to  reduce  workload  for  each  task. 

The  C^  prototype  usability  portion  of  this  same  questionnaire  was  designed  to  provide  the 
software  designers  more  explicit  feedback  on  desired  improvements  to  the  prototype.  Players 
were  asked  to  provide  a  more  detailed  assessment  of  the  prototype’s  usefulness  based  on:  task 
clarity,  task  difficulty,  display  characteristics,  ease  of  accomplishing  key  selected  tasks,  and  ease 
of  understanding  the  information  provided  in  the  prototype’s  various  display  windows.  A  final 
section  asked  the  players  to  assess  current  and  desired  prototype  default  settings  (e.g.,  search 
radius  for  LAM/PAM,  saving  adjustments  made  to  manipulated  sensor  images).  The  set  of 
default  settings  identified  in  the  questionnaire  were  based  on  perceived  user  problems  during 
ARI’s  HCI  analysis  for  Experiment  2.  This  questionnaire  was  administered  at  the  completion  of 
Run  5. 


Training.  A  training  questionnaire  was  administered  at  the  completion  of  Run  1  to 
determine  if  the  players  felt  that  the  formal  and  informal  training  the  command  group  received 
was  adequate  for  performing  as  a  member  of  the  Unit  Cell.  The  questionnaire  focused  primarily 
on  the  adequacy  of  the  training  content  and  the  adequacy  of  training  time  for  both  individual  and 
collective  performance. 


Analysis 

The  analysis  and  results  provided  here  are  limited  to  descriptive  statistics.  Inferential 
statistics  were  considered  inappropriate  due  to  limitations  in  experimental  design  and  execution, 
and  the  small  (n  =  4)  sample  of  subject  players.  Particularly,  the  manipulation  of  Complexity 
level  across  runs  was  confounded  with  run  order  effects.  In  addition,  player  personnel  in 
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Experiment  3  varied  across  runs.  Alternate  personnel,  including  a  Major  and  an  Reserve 
Officers’  Training  Corps  (ROTC)  cadet,  were  required  for  the  majority  of  runs  as  substitutes  for 
at  least  some  of  the  four  primary  Active  duty  Lieutenant  Colonels.  Alternate  personnel  had 
substantial  training  on  C^  prototype,  and  had  performed  at  least  pilot  exercise  runs. 

Descriptive  statistics  were  also  warranted  given  the  exploratory  and  iterative  nature  of  the 
FCS  C^  program.  The  capabilities  of  the  C^  prototype  and  its  integration  with  supporting 
simulation  were  incomplete  and  unreliable.  Partial  and  system-wide  software  crashes  disrupted 
the  execution  of  experimental  runs  and  on  occasion  resulted  in  less  than  graceful  degradation  of 
the  supporting  command  and  control  environment. 

Results 

Results  are  organized  in  the  following  manner:  Verbal  Interactions  during  execution  and 
then  planning,  HCI  during  execution  and  then  planning,  and  Subjective  Measures. 

Results  on  verbal  and  HCI  interactions  will  focus  primarily  on  the  execution  phase. 
Results  from  the  planning  phase  are  provided  after  the  execution  phase.  A  rationale  for  this 
somewhat  reverse  order  of  presentation  seems  in  order.  First,  prior  FCS  C^  experiments  and 
reports  focused  almost  exclusively  on  the  execution  phase  versus  planning.  For  program 
consistency  and  priority  of  ARI’s  analytic  effort,  therefore,  a  primary  focus  on  execution  was 
maintained.  Second,  the  data  collected  and  results  reported  on  planning  are  regarded  as 
incomplete  and  preliminary  findings. 

The  fidelity  of  the  verbal  recordings  during  planning  was  not  high.  Some  planning 
communications  were  not  collected  due  to  inconsistent  use  of  headsets  in  the  C^  vehicle,  and 
discussions  outside  the  C^  vehicle.  Moreover,  planning  activities  were  often  limited,  as  the 
players  had  performed  very  related  planning  on  prior  runs.  In  particular,  planning  with  respect  to 
Terrain  was  curtailed.  Prior  experience  with  the  National  Training  Center  (NTC)  terrain 
database  included  not  only  the  participants  30+  experimental  runs  during  Experiments  1-3,  but 
also  numerous  live  rotations  at  the  NTC  during  their  Army  careers.  Players  routinely  re-used 
many  of  the  same  graphic  control  measures  (e.g.,  phase  lines,  axes  of  attacks,  and  named  areas  of 
interest)  during  planning  that  they  had  created  and  stored  in  their  C^  prototypes  for  earlier  runs. 

Verbal  Interactions — ^Execution  Phase 

Run  Characteristics.  A  brief  review  of  key  run  characteristics  is  provided.  Verbal 
commrmications  were  transcribed,  coded  and  analyzed  to  identify  categories  and  patterns  of 
commimication,  as  described  in  the  Method  section.  Table  6  characterizes  Runs  10  and  1 1 
according  to  duration,  cumulative  amovuit  of  silence,  and  number  of  verbal  chunks.  Both  runs 
lasted  approximately  1.5  hours.  During  the  runs,  verbal  interactions  were  almost  continuous. 
Cumulative  time  in  silence,  counting  silences  lasting  3  seconds  or  more,  was  under  3.5  minutes 
or  less  than  4%  of  total  run  time. 
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Table  6 


Key  Characteristics  for  Execution  Phases,  Runs  1 0  and  1 1 


Complexity 

Too  High 

High 

Rim  Number 

10 

11 

Run  Duration 

84  minutes 

89  minutes 

Cumulative  Silence* 

3.4  minutes 

2.9  minutes 

#  Verbal  Chunks 

436 

461 

*Timing  initiated  after  3  seconds  of  silence. 

Communication  by  Source.  Figme  2  displays  the  percentage  of  verbal  communication  by 
Source.  Source  data  identifies  the  participants  engaged  in  communication  for  each  verbal  chunk. 
As  expected  based  on  prior  experiments,  the  vast  majority  of  commimications  were  internal  to 
the  command  group  players  (Within  Cell-Black).  Remaining  verbalizations  were  fairly  evenly 
divided  among  chunks  in  which  the  command  group  spoke  with  superiors  (Black/Blue), 
subordinates  (Black/Subordinates),  and  support  personnel  and  observers  in  the  C^  vehicle 
(Other).  These  Source  results  help  clarify  that  the  command  group’s  almost  continuous  stream 
of  communication  was  predominantly  with  one  another.  Notably,  this  pattern  of  steady 
verbalization  occurred  in  the  context  of  (a)  the  command  group’s  access  to  a  visually  rich  and 
timely  depiction  of  the  battlefield  on  their  C^  prototype,  and  (b)  ongoing  interactions  with  their 
C9  prototypes  to  command  and  control  a  predominantly  robotic  force,  as  reported  later  in  the 
HCI  Analysis  section. 


Within  Cell  Black/  Subordinate  Black/  Blue  Other 

Source 

Figure  2.  Percent  of  Verbalization  by  Source,  Runs  10  and  1 1  Execution. 

Communication  by  Function.  Figure  3  depicts  the  percent  of  verbal  communication  by 
Function  during  execution,  for  Runs  10  and  1 1.  Communications  related  to  the  See  function 
were  the  most  frequent,  about  30%  of  all  verbalizations.  The  next  most  fi’equent  C  functions 
discussed  were  Strike  and  Move,  followed  by  BD A.  Comparing  across  runs,  the  relative 
percentage  of  communications  by  C^  function  was  remarkably  similar.  Moreover,  the  relative 
percentage  of  communications  by  C^  function  was  similar  to  that  observed  in  Experiment  2. 


Plan  Move  See  Strike  BDA  Other 

Function 


Figure  3.  Percent  of  Verbalization  by  Function,  Runs  10  and  1 1  Execution. 

Communication  by  Valence.  Valence  ratings  were  assigned  to  verbal  behaviors  in  order 
to  distinguish  communications  conveying  positive  versus  negative  status  on  accomplishing  basic 
functions  and  tasks  within  those  factions,  as  described  in  the  Method  section.  Results  on 
Valence  by  Function  are  summarized  in  Figure  4  (Run  10)  and  Figure  5  (Run  1 1).  Overall,  by 
far  the  majority  of  verbal  communications  were  rated  positively  in  terms  of  Valence.  This 
pattern  of  predominantly  positive  status  on  accomplishing  Functions  was  found  for  See,  Plan, 

Move,  and  Strike  related  communications.  In  contrast,  BDA  and  Other  related  communications 
were  relatively  more  negative.  For  BDA,  percentage  of  positive  versus  negative  verbalizations 
was  nearly  equivalent.  For  “Other”  communications,  particularly  technical  status  comments, 
communications  were  predominantly  negative.  While  the  Other  communications  do  not  directly 
relate  to  Fxmctions,  they  provide  useful  information  on  player  perceived  technical  limitations 
in  the  prototype  (e.g.,  slow  processing  and  system  crashes)  that  may  have  undermined  the 
command  group’s  performance  across  functions. 


Plan  See  Move  Strike  BDA  j  Other  j 


Quartiie  and  Function 

Figure  4.  Percent  of  Verbalization  by  Function,  Valence  and  Quartiie,  Run  10  Execution. 
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Figure  5.  Percent  of  Verbalization  by  Function,  Valence  and  Quartile,  Run  1 1  Execution. 

Negatively  valenced  communications  provide  useful  diagnostic  information  on  the 
c^abilities  of  the  prototype.  Efforts  by  the  FCS  program’s  Technical  Team  to  refine  the 

C^prototype  should  attend  closely  to  the  problems  discussed  in  negatively  rated  verbalizations. 
For  example.  Figure  4’s  quartiles  help  pinpoint  when  negative  verbalizations  occurred  in  Run  10. 
During  quartile  2  of  Run  10,  player  verbalizations  indicated  problems  with  Move  and  BDA 
fimctions.  These  communications  stress  the  difficulty  of  controlling  the  micro-UAVS  that 
resulted  in  the  loss  of  two  of  the  four  allotted  the  Unit  Cell.  For  BDA,  negatively  valenced 
communications  indicated  the  participants’  inability  to  kill  designated  targets.  During  Quartile  4 
of  Rxm  10,  negative  Move  related  comments  stressed  the  inability  to  maneuver  several  platforms, 
including  Future  Warriors,  Robo-Scout  ground  surveillance  radar  (GSR),  and  LOS.  In  fact.  Run 
10  was  terminated  about  the  time  all  systems  became  “frozen.”  Notably,  the  84-minute  duration 
of  Run  10  allowed  the  Unit  Cell  sufficient  time  to  complete  their  mission. 


Communications  by  Type.  Type  categorizations  were  designed  to  help  identify  the 
purpose  of  the  command  group’s  communications.  Figure  6  depicts  the  percentage  of  verbal 
interactions  by  Type  for  Run  10  and  1 1 .  The  distributions  by  Type  are  fairly  similar  across  runs. 
The  primary  types  of  communications  were  Share  and  Ask  that  together  accounted  for  over  70% 
of  the  verbalizations  in  each  run.  The  relative  dominance  of  Share  and  Ask  communications 
appeared  to  underscore,  respectively,  the  collaborative  nature  of  the  command  group’s  work,  and 
a  fair  degree  of  uncertainty  about  the  unfolding  battlefield  situation.  The  pattern  of  distributions 
by  Type  is  also  similar  to  patterns  observed  in  Experiment  2. 
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Share  Action  Direction  Ask  Process  Decide  Other 

Type 


Figure  6.  Percent  of  Verbalization  by  Type,  Runs  10  and  1 1  Execution. 

Communications  by  METT-TC  Factors.  Figure  7  shows  the  frequency  of  verbal 
interactions  by  METT-TC  Factor  (Mission,  Enemy,  Terrain,  Troops,  Time,  Civilians)  for  Runs 
10  and  11.  Troops  and  Enemy  related  communications  virtually  monopolized  verbal  interaction, 
together  accounting  for  over  90%  of  the  chunks.  Recall,  METT-TC  Factors  were  further 
analyzed  into  25  sub-factors.  For  example.  Enemy  categories  included  4  sub-factors  (Location, 
Identification,  Disposition,  and  BDA).  Figure  8  shows  the  distribution  of  Enemy  related 
communications  by  sub-factor.  Almost  80%  of  Enemy  related  communications  concerned 
Location  or  BDA,  with  approximately  equal  amounts  of  discussion  devoted  to  each.  This  pattern 
may  indicate  the  command  group’s  collaborative  and  balanced  efforts  to  “find  and  fix”  Enemy 
elements. 


Mission  Enemy  Terrain  Troops  Time 

METT-TC  Factor 


Figure  7.  Percent  of  Verbalization  by  METT-TC  Factor,  Runs  10  and  1 1  Execution. 

Note:  Civilians  =  0%  in  Runs  1 0  and  1 1 ;  Other  =  1 .6%  and  2.6%,  Run  1 0  and  1 1  respectively. 
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Figure  8.  Percent  of  Verbalization  by  Enemy  Sub-Factor,  Runs  10  and  1 1  Execution. 

Similarly,  Troops  related  commimications  were  categorized  into  14  sub-factors,  and  their 
distribution  is  shown  in  Table  7  with  selected  sub-factors  shown  in  Figure  9.  As  in  Experiment 
2,  the  most  frequent  Troops  sub-factor  was  Strike-Lethal  communications,  related  to  laimching, 
firing,  and  deploying  with  intent  to  destroy  (e.g.,  LAMs).  The  next  most  frequent  Troops  sub¬ 
factor  was  Move  with  verbalizations  related  to  moving,  managing,  and  maneuvering  Unit  Cell 
assets.  Finally,  there  was  a  notable  decrease  in  complaints  about  the  prototype  Information 
Technology  (IT)/Commander’s  Support  Environment  (CSE)  in  Experiment  3,  compared  to 
Experiment  2.  This  decrease  seems  to  suggest  the  command  group  experienced  fewer  technical 
problems  with  their  prototypes  and/or  a  more  stable  operational  environment  during 
Experiment  3. 


Table  7 

Percent  of  Verbalization  by  Troops  Sub-Factors,  Runs  10  and  1 1  Execution 


Troops  Factor 

Percentage  (Run  10) 

Percentage  (Run  11) 

Position 

1.8 

1.5 

Mobility 

1.0 

1.1 

Sensors 

10.2 

12.6 

Strike  Ability 

3.2 

4.1 

Commimications 

5.6 

0.3 

IT/CSE 

9.8 

4.8 

Caution 

6.0 

2.9 

Sundvability  Move 

_ 

1.5 

Loss 

2.5 

1.8 

Move 

25.3 

28.5 

Strike-Lethal 

30.6 

34.4 

Strike-Nonlethal 

0.3 

2.6 

Training 

0 

0 

Other 

1.0 

3.7 
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Figure  9.  Percent  of  Verbalization  by  Selected  Troops  Sub-Factors,  Runs  10  and  1 1  Execution. 

Command  Considerations.  Figure  10  illustrates  the  frequency  of  communications  for 
each  of  the  nine  categories  of  Command  Considerations.  These  data  are  presented  in  terms  of 
absolute  frequency,  rather  than  percentages,  given  the  exploratory  nature  of  this  analysis.  The 
art’s  initial  concern  was  if,  and  how  often,  such  considerations  occurred.  Table  8  provides 
examples  of  communication  segments  coded  by  Command  Considerations.  Commxmications 
related  to  all  nine  Command  Considerations  were  found  in  each  run.  This  finding  was  expected 
for  the  expert  command  group  in  Experiment  3,  if  the  cognitive  considerations  identified  are 
required  for  battle  command.  For  comparisons  with  less  experienced  participants,  such  as 
Cadets,  Command  Considerations  may  prove  more  discriminating  and  useful.  The 
Comprehensive  Report  by  ART  across  all  four  (4)  FCS  C^  experiments  will  include  a  section  on 
Cadet  performance  fi-om  the  FCS  C^  Summer  Study. 


oRun  10 
H  Run  1 1 


0  2  4  6  8  10  12 


Observed  Frequency 


Figure  10.  Frequency  of  Command  Considerations,  Runs  10  and  1 1  Execution. 
Note:  Command  Considerations  defined  in  Table  2. 
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Table  8 


Examples  of  Verbal  Communications  by  Command  Considerations 


Consideration 

Example 

Keep  sight  of  the  big 
picture  and  mission 
intent. 

I  think  they  are  probably  dead,  or  mobility  kills  which  for 
this  scenario  is  both  good. 

Yeah. 

We  won't  waste  any  more  rounds  on  him. 

Yeah.  Actually,  just  as  long  as  we  hit  him  even  if  they  are 
fire  power  kills,  I  don't  care  less. 

Jack,  what  we  don't  want  to  do  is  get  into  the  Netfires. 

Yeah  exactly,  exactly. 

So  that’s  why  I'll  leave  the  troop  transport  there,  to  protect 
our  flank. 

Protect  the  flank. 

Model  a  thinking  enemy. 

That  means  he  is  moving  out  of  sector. 

I  mean  they  dropped  it  right  on  top  of  him. 

Either  that  or  he  is  trying  to  reposition  from  where  they  told 
us  he  was  at  Start  Ex. 

That  could  very  well  be. 

Right  in  the  middle  of  the  valley,  back  to  a  ridgeline,  or 
forward  into  the  gap  there. 

Consider  effects  of 
terrain/time. 

I'm  just  looking,  I'm  trying  to  find  that  freaking  keyhole  up  in 
here. 

Continual  situational 
assessment  and 
dynamic/contingency 
planning. 

I  stopped,  go  to  the  heads  up.  I  stopped  him,  that's  his  eyes 
so  if  this  guy  continues  in  this  direction  that's  a  keyhole. 

And  if  he  ends  up  here  I  ought  to  be  able  to  see  him  and  that 
would  be  v^thin  Javelin  range. 

Okay. 

If  he  pops  up  in  any  of  these  areas  here. 

Use  all  assets  available. 

Brooks,  I  don't  think  we  are  going  to  get  the  A 160  any  time 
soon,  so  I  need  you  “Sir”  to  get  the  GSR  up  on  some  higher 
ground  there  right  in  front  of  you  and  get  a  mast  up. 

Create  synergistic  effects 
wdth  multiple 
assets/teamwork. 

Hey  Jack,  head's  up  chief,  here's  my  second  IFV  team  in  the 
South. 

Okay. 

It  runs... 

Now  that  tank. 

Yeah  we  got  to  kill  him.  He  dies  first. 

But  he  runs  here,  this  is  pretty  much  covering  the  meadow 
this  point  right  here  is  where  he  stops... infantry  North. 

But  this  time,  all  hell  is  going  to  break  loose. 

Table  Continues 
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All  of  a  sudden  two  different  infantry  teams  are  popping  up 
on  the  ridge,  well  one  on  the  ridge  and  one  on  the  road,  but 
they  are  only  300m  apart.  All  of  a  sudden  he's  got  clear 
visibility  across  the  valley. 

Very  nice,  now  which  one,  that's  the  IFV. 

That's  the  IFV  with  2  mounted  teams. 

Okay. 

Now  I've  got  the  dismounts  starting  off,  2  routes,  this  guy  is 
here,  he  moves  here,  and  all  of  a  sudden. . . 

That's  a  pretty  good  view. 

He  gets  a  good  view  out  of  the  valley.  This  bobo  down  here. 
He  is  going  to  move  doivn  this  route,  and  he  should  be  able 
to,  shortly,  acquire  anything  out  in  that  open  area  there,  at 
this  position.  See  where  that  is  open? 

Oh  yeah. 

So  that's  right  now  his  route  plan. 

For  dismounts? 

Yeah. 

Battlefield  visualizations 
that  are  d5mamic/ 
predictive/proactive. 

Hey  Dave,  if  I  was  a  guessing  man,  I  would  say  that  radio 
link  3,  which  is  up  there  where  that  PAM  lost  comms,  has 
now  become  unknown  27.  I'm  just  telling  you,  I  think  that 
very  well  may  have  moved  that  down  there. 

Make  information 
requirements  known. 

I  want  to  see  in  front  of  us  with  those  Micro-UAVs,  it  takes  a 
long  time  to  try  to  develop  what's  in  front  of  us,  so  we  need 
to  do  that  as  quickly  as  possible. 

Execution  is  self-initiated 
and  preceded  by  plan 
refinement/coordination. 

No  other  detections,  Micro-UAVs  going  out  there  taking 
pictures. 

Go  micro  go. 

Got  one  back  here.  Micro  1  waiting  for  its  detections  up 
there  in  the  North,  to  go  and  take  pictures  there.  The  other  3 
are  out  on  the  route. 

Verbal  Interactions — Planning  Phase 

This  section  provides  results  of  the  verbal  interactions  by  the  command  group  during  the 
planning  phase.  The  introductory  discussion  of  Planning  Characteristics  stresses  the  incomplete 
and  preliminary  nature  of  planning  results. 

Planning  Characteristics.  Verbal  commimications  during  planning  were  transcribed, 
coded  and  analyzed  to  identify  categories  and  patterns  of  commimication,  as  described  in  the 
Method  section.  Table  9  characterizes  the  two  planning  phases  analyzed  according  to  their 
duration,  cumulative  amount  of  silence,  and  number  of  verbal  chunks.  As  indicated  by  duration 
times,  planning  phases  were  notably  shorter  than  the  execution  phases.  During  the  execution 
phase  for  each  run,  verbal  commimications  occurred  96%  of  run  time.  In  the  planning  phase, 
however,  verbal  communications  occurred  during  86%  of  Run  10  and  75%  of  Run  11.  Possible 
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explanations  for  less  verbal  interaction  during  planning  are  that  not  all  participants  remained  in 
the  vehicle,  and  headsets  were  not  continuously  wore  during  the  entire  planning  phase. 

Table  9 

Key  Characteristics  for  Planning  Phases,  Runs  10  and  1 1 


Run  Number 

10 

11 

Run  Duration 

56  minutes 

39  minutes 

Cumulative  Silence* 

7.6  minutes 

10.1  minutes 

#  Verbal  Chunks 

75 

61 

*Timing  initiated  after  3  seconds  of  silence. 


A  more  interesting  and  important  explanation  may  be  actual  differences  in  the 
communication  requirements  for  planning  versus  execution.  For  example,  players  appeared  to 
spend  more  time  entering  duty-specific  mission  data  (e.g.,  routes  and  tasks)  into  their  C 
prototypes  during  planning.  However,  method  shortcomings  should  be  eliminated  or  at  least 
reduced  before  more  firm  conclusions  on  actual  differences  are  reached.  Moreover,  given  the 
smaller  sample  of  verbal  communications  during  planning,  more  variation  in  planning  results 
should  be  expected  and  less  confidence  should  be  placed  in  apparent  differences.  In  sum,  ARI’s 
emphasis  on  the  exploratory  nature  of  planning  results  is  stressed. 

Communication  by  Source.  Figure  1 1  displays  the  percentage  of  verbal  communications 
during  planning,  by  Source.  Source  data  identifies  the  participants  engaged  in  communication 
for  each  verbal  chunk.  Similar  to  the  execution  phase,  the  majority  of  communications  were 
internal  to  the  command  group  players.  Within  Cell  (Black).  Remaining  verbalizations  were 
evenly  divided  among  chunks  in  which  the  players  spoke  with  superiors  (Black/Blue), 
subordinates  (Black/Subordinates)  and  support  personnel  and  observers  in  the  vehicle  (Other). 

Communication  by  Function.  Figure  12  provides  the  percent  of  verbal  communications 
by  Function  during  planning  for  Runs  10  and  1 1 .  For  both  runs.  Plan  related  communications 
were  the  most  frequent  category  by  function,  almost  50%  of  all  verbalizations.  This  high 
proportion  of  Plan  related  communications  is  hardly  surprising,  as  the  focus  of  these  sessions 
was  planning,  but  it  indicates  the  potential  for  basic  differences  between  planning  and  execution 
phases.  The  next  most  frequent  planning  communication  category  was  “Other”  for 
verbalizations  not  matching  any  of  the  pre-defined  set  of  functions.  Overall,  the  results 

suggests  that  most  functions  were  addressed  during  planning,  but  more  careful  examination 
may  require  revision  of  the  methods  and  codes  used  to  assess  planning  verbalizations, 
particularly  the  Other  category. 
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Within  Cell  Black/  Subordinate  Black/  Blue  Other 

Source 

Figure  1 1 .  Percent  of  Verbalization  by  Source,  Runs  10  and  1 1  Planning. 


Plan  Move  See  Strike  BDA  Other 

Function 

Figure  12.  Percent  of  Verbalization  by  Function,  Runs  10  and  1 1  Planning. 

Communication  by  Valence.  Recall,  Valence  codes  were  assigned  to  verbal  behaviors  in 
order  to  distinguish  communications  conveying  positive  versus  negative  status  on  accomplishing 
Functions  and  supporting  tasks.  Results  on  Valence  by  Function  are  summarized  in  Figure  13 
for  Runs  10  and  11.  Overall,  by  far  the  majority  of  verbal  communications  were  rated  positively 
in  terms  of  Valence.  Most  problems  in  accomplishing  functions  during  planning  were 
associated  with  Plan  and  See  functions  during  Run  1 1 . 
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Figure  13.  Percent  of  Verbalization  by  Function  and  Valence,  Runs  10  and  1 1  Planning. 


Communications  by  Type.  Type  categorizations  were  designed  to  help  identify  the 
purpose  of  the  command  group’s  communications.  Figure  14  depicts  the  percentage  of  verbal 
interactions  during  planning  by  Type,  for  Rrms  10  and  1 1 .  As  with  execution,  the  primary  types 
of  communications  were  Share  and  Ask  that  together  accounted  from  50%  (Run  10)  to  70% 
(Run  11)  of  all  planning  verbalizations.  Again,  the  relative  dominance  of  Share  and  Ask 
communications  appears  to  underscore,  respectively,  the  collaborative  nature  of  the  command 
group’s  work,  and  a  fair  degree  of  imcertainty  about  the  upcoming  battlefield  situation. 


□  Run  10 
m  Run  1 1 


Type 


Figure  14.  Percent  of  Verbalization  by  Type,  Runs  10  and  1 1  Planning. 

Communications  by  METT-TC  Factors.  Figure  1 5  shows  the  frequency  of  verbal 
interactions  by  METT-TC  Factor  for  Runs  10  and  1 1.  Across  both  runs.  Troops  related 
communications  were  the  most  frequently  discussed  topic  during  planning.  However,  during 
Run  10,  communications  related  to  Time  (52%)  were  the  single  most  discussed  METT-TC 
Factor  during  planning.  In  addition.  Enemy  and  Mission  related  communications  also  dominated 
planning  conversations.  For  example,  during  Run  10,  25%  of  all  planning  commumcations  were 
Mission  related,  indicating  the  players’  concerns  with  mission  goals  and  plans  prior  to  execution. 
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□  Run  10 

□  Run  1 1 


METT-TC  Factor 


Figure  15.  Percent  of  Verbalization  by  METT-TC,  Runs  10  and  1 1  Planning. 

Table  10  and  Figure  16  provide  a  closer  look  at  Troops  related  communications  by  sub¬ 
factors.  Strike  Ability  and  Strike-Lethal  were  dominant  discussion  topics  and  underscored  the 
command  group’s  concerns  with  launching,  firing,  and  deploying  lethal  effects.  Notably,  a  “hot” 
topic  under  Troops  assets  was  the  prototype  (IT/CSE)  that  accounted  for  3 1%  of  all  Troops 
related  communications  during  Run  10  planning,  and  25%  during  Run  1 1  plarming. 


Table  10 


Percent  of  Verbalization  by  Troops  Sub-Factors,  Run  10  and  1 1  Planning 


Troops  Factor 

Percentage  (Run  10) 

Percentage  (Run  11) 

Position 

0 

7.1 

Mobility 

0 

0 

Sensors 

0 

0 

Strike  Ability 

34.4 

0 

Commimications 

6.9 

21.4 

IT/CSE 

31.0 

25.0 

Caution 

0 

0 

Survivability  Move 

0 

0 

Loss 

0 

0 

Move 

24.1 

17.8 

Strike-Lethal 

34.1 

14.3 

Strike-Nonlethal 

3.4 

3.6 

Training 

0 

0 

Other 

6.9 

10.7 

Verbalizations  about  the  prototype  were  predominantly  centered  on  how  to  correctly 
input  operational  plans.  In  particular,  players  expressed  uncertainty  about  what  could  be  done 
with  the  prototype  in  planning  versus  execution  phases.  Given  these  were  the  last  runs  during 
Experiment  3,  verbalizations  about  the  prototype  indicate  that  training,  particularly  in  support 
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of  planning,  was  inadequate.  Understanding  the  complexities  of  the  prototype,  particul^Iy 
planning  inputs  and  parameters  related  to  automated  See,  Move,  and  Strike  functions,  requires 
structured  training  directed  at  these  functions  with  clear  operational  feedback. 


Troops  Sub-Factor 

Figure  16.  Percent  of  Verbalization  by  Selected  Troops  Sub-Factors,  Runs  10  and  1 1  Planning. 

Command  Considerations.  Figure  17  illustrates  the  frequency  of  communications  for 
each  of  the  nine  categories  of  Command  Considerations  during  planning  for  Runs  10  and  1 1.  As 
with  execution,  these  data  are  presented  in  terms  of  absolute  frequency,  rather  than  percentages, 
given  the  exploratory  nature  of  this  analysis.  Communications  related  to  nearly  all  Command 
Considerations  occurred  during  the  planning  phases. 


Two  near  exceptions  included  the  relatively  low  frequencies  tabulated  for  Coordinate 
(create  synergistic  effects)  and  Assets  (use  all  assets  available).  However,  with  experienced 
command  groups,  the  coordinated  use  of  all  assets  may  be  more  implied  than  explicit. 
Comparisons  with  less  experienced  participants  are  recommended.  Such  comparisons  may 
identify  interesting  differences  between  experts,  intermediates  and  novices  and  provide  an 
empirical  basis  for  training  and  evaluating  Command  Considerations. 
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Figure  17.  Frequency  of  Command  Considerations,  Planning  Runs  10  and  1 1  Planning. 
Note:  Command  Considerations  defined  in  Table  2. 
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Human-Computer  Interactions — Execution  Phase 

The  primary  measures  of  performance  for  assessing  HCI  were  task  frequency  and 
duration  by  Function  and  Sub-Function.  Overall,  the  results  provide  a  detailed  account  of  the 
human-computer  interactions  required  to  command  and  control  the  Unit  Cell  during  Run  10 
execution.  Results  indicated  the  frequency,  duration  and  type  of  HCI  tasks  differed  across  the 
command  group,  and  across  left  versus  right  screen  performance  at  each  duty  position. 
Performance  time  and  errors  were  less  useful  as  performance  criteria.  Performance  times  for 
most  interactions  were  typically  5  seconds  or  less,  and  did  not  appear  to  be  associated  with  any 
other  indicator  of  success  or  failure.  However,  such  inferences  are  limited  by  the  present  focus 
on  only  one  experimental  run. 

Time  Duration  for  HCI.  Time  duration  for  task  completion  was  examined  as  an 
important  aspect  of  HCI  performance.  Tasks  requiring  longer  performance  times  could  be 
candidates  for  redesign,  machine  aiding,  or  additional  training.  Task  performance  time  was 
estimated  by  identifying  Start  and  Stop  actions,  typically  involving  the  selection  of  a  menu 
option.  Overall,  during  the  execution  phase  for  Run  10  the  command  group  players  performed  a 
total  of  1,043  human-computer  interactions.  Approximately  13.5%  of  these  interactions  (141  of 
the  1,044)  required  more  tiian  5  seconds  to  perform.  Table  1 1  provides  frequency  data  and 
performance  times  in  seconds,  mean  and  standard  deviation  (SD),  for  these  long  duration  tasks. 
Notably,  the  same  task  types  listed  in  Table  1 1  were  sometimes  more  qmckly  performed, 
required  less  than  5  seconds.  Accordingly,  Table  1 1  reports  only  the  subset  of  interactions  for 
the  tasks  listed  that  took  longer  than  5  seconds  to  perform. 

Table  11 

HCI  Long  Duration*  Tasks  by  Frequency  and  Time,  Run  10  Execution 


Human-Computer 
Interaction  Task 

Frequency 

Time  in 
Seconds 
(Mean/SD) 

Commander 

Battlespace 

Information 

Effects 

Create  Ground  Route 

— 

11 

— 

— 

17.6/9.4 

Create  Air  Route 

— 

1 

18 

— 

15.1/9.8 

Recognize  Targets 

1 

— 

5 

— 

17.0/12.6 

Assess  Battle  Damage 

8 

24 

67 

4 

18.2/12.7 

Crash/Reboot 

— 

2 

— 

— 

51.5/27.6 

*Long  duration  tasks  requiring  more  than  five  (5)  seconds  to  perform. 


Success  Criteria  for  HCI  Task  performance  “success”  was  not  a  useful  performance 
criterion  for  Experiment  3.  There  was  little  discemable  evidence  that  command  group  players 
made  serious  mistakes  in  performing  HCIs,  and  no  clear  indication  that  errors  impacted  Unit  Cell 
performance.  The  only  tasks  associated  with  clearly  observable  errors  were  eleven  (11) 
inadvertent  clicks  on  Graphic  Control  Measures  (GCMs). 
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Frequency  ofHCI  by  Duty  Position  and  Function.  Figure  1 8  presents  the  frequency  of 
interactions  performed  at  each  player’s  workstation  by  Function.  Across  player  workstations, 
the  greatest  number  of  interactions  performed  supported  the  ability  to  See  the  battlefield.  In 
order  by  amount,  the  percentage  of  See  related  interactions  performed  by  each  player  was 
Commander  (96.2%),  Information  Manager  (89.9%),  Effects  Manager  (64.6%),  and  Battlespace 
Manager  (60.3%).  Interactions  supporting  Strike  also  represented  a  large  proportion  of  the  total 
tasks  performed,  particularly  for  the  Battlespace  Manager  (29.0%),  and  Effects  Manager 
(35.4%).  Much  smaller  proportions  of  Move  related  interactions  occurred,  primarily  by  the 
Battlespace  Manager  (9.0%),  and  Information  Manager  (8.0%).  Only  one  Plan  related 
interaction  (performed  by  the  Battlespace  Manager)  was  identified,  involving  the  use  of  the 
overhead  display  to  show  the  location  and  sensor  reach  capability  of  the  Roboscout  vehicle.  To 
clarify  the  presentation  of  results,  this  single  instance  of  a  Plan  related  interaction  is  not  included 
in  the  figures,  tables  or  discussion  ofHCI  during  the  execution  phase. 


mMove 

□  See 

□  Strike 


Figure  18.  Frequency  ofHCI  by  Duty  Position  and  C^  Function,  Run  10  Execution. 

Frequency  ofHCI  by  Sub-Function  and  Duty  Position.  Table  12  provides  frequency  data 
by  C^  Function  and  the  12  Sub-Fvmctions  performed  during  Run  10  execution.  Notably,  no  HCI 
tasks  were  performed  on  the  remaining  five  of  seventeen  Sub-Functions  depicted  in  Figure  1. 
Sub-Functions  not  performed  were  Create  Mission/COA  and  Alerts  under  Plan,  Create  Group 
Follow  under  Move,  and  Schedule  Fires  and  Create/Modify  AGM  under  Strike.  The  complete 
set  ofHCI  frequency  data  by  duty  position  during  Run  10  execution  is  provided  in  Appendix  F. 


Across  all  player  workstations,  the  most  frequent  interactions  were  performed  on  See 
related  tasks.  The  most  frequent  interactions  by  the  Commander  and  Battlespace  Manager  were 
Display  Sensor  Data,  respectively  45  and  66  interactions.  Display  Sensor  Data  interactions 
primarily  involved  moving  the  cursor  over  enemy  and  friendly  icons  to  “pop”  up  a  window  that 
temporarily  displayed  information  about  that  icon  (e.g.,  type  of  target,  time  when  detected,  and 
t5q)e  of  sensor  that  detected  target). 


The  most  frequent  interactions  for  the  Information  Manager  were  Assess  Battle  Damage 
(132  interactions).  This  interaction  requires  opening  the  Enemy  Intelligence  window  and 
reviewing  a  sensor  picture  to  determine  the  extent  of  damage  to  the  pictured  vehicle,  and  then 
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updating  the  target  image  information  to  reflect  its  tj'pe  and  status.  Most  frequent  interactions 
performed  by  the  Effects  Manager  involved  Manipulate  Map  tasks,  which  included  changing  the 
magnification  of  the  map,  and  scrolling  the  display  to  present  different  areas  of  the  map. 

Table  12 

Frequency  and  Percent  of  Human-Computer  Interactions  by  Duty  Position 


Function 

Subfunction 

Commander 

Freq.  % 

Battlespace 
Manager 
Freq.  % 

Information 
Manager 
Freq.  % 

Effects 
Manager 
Freq.  % 

Move 

— 

28 

9.1 

27 

8.0 

— 

— 

Move  Ground  Assets 

— 

— 

28 

9.1 

— 

— 

— 

— 

Move  Air  Assets 

— 

— 

— 

— 

27 

8.0 

— 

— 

See 

101 

96.2 

189 

61.2 

304 

89.9 

188 

64.6 

Manipulate  Map 

11 

10.5 

18 

5.8 

6 

1.8 

91 

31.3 

Use  Visualization  Tools 

6 

5.7 

29 

9.4 

26 

7.7 

12 

4.1 

Display  Sensor  Data 

45 

42.9 

66 

21.4 

102 

30.2 

62 

21.3 

Recognize  Targets 

1 

1.0 

2 

0.6 

37 

10.9 

1 

0.3 

Assess  Battle  Damage 

9 

8.6 

42 

13.6 

132 

39.1 

4 

1.4 

Summarize  Situation 

29 

27.6 

32 

10.4 

1 

0.3 

18 

6.2 

Awareness  (SA) 

Strike 

4 

3.8 

90 

29.1 

7 

2.1 

103 

35.4 

New  Features 

— 

— 

1 

0.3 

— 

— 

19 

6.5 

Designate  Target 

— 

— 

41 

13.3 

1 

0.3 

28 

9.6 

Fire  Weapon 

— 

— 

46 

14.9 

1 

0.3 

32 

11.0 

Monitor  Fires 

4 

3.8 

2 

0.6 

5 

1.5 

24 

8.2 

Other 

— 

— 

2 

0.6 

— 

— 

— 

— 

Total 

105 

100 

309 

100 

338 

100 

291 

100 

Move  Function.  During  Run  10  execution,  55  of  the  1,043  interactions  (5.3%)  were 
performed  to  move  assets  of  the  Unit  Cell.  To  move  ground  assets,  the  Battlespace  Manager 
performed  a  total  of  28  interactions.  These  interactions  involved  deleting  previously  entered 
routes  and  entering  new  route  points  by  clicking  locations  on  the  map.  Platform  moves  entailed 
selecting  menu  options  to  start,  halt,  or  resume  their  planned  routes.  Movement  of  air  assets 
involved  the  performance  of  27  separate  interactions  by  the  Information  Manager  to  select 
reconnaissance  targets  and  areas.  Air  asset  movement  routes  were  created  by  either  clicking  on  a 
target  or  area  icon,  or  by  selecting  a  target  or  area  from  a  menu. 

See  Function.  The  majority  (75. 1  %)  of  interactions  performed  supported  the  See 
function,  and  involved  managing  the  display  of  map  and  sensor  assets.  Table  12  shows  that 
Manipulation  Map  tasks  (i.e..  Zoom  and  Scroll  interactions)  were  performed  frequently  by  the 
Effects  Manager  (91  interactions),  but  less  frequently  by  the  Cell  Commander,  Battlespace,  and 
Information  Managers.  The  Plot  Intervisibility  tool  was  not  used  in  Run  10,  however,  the  Heads 
Up  Display  (HUD)  was  used  by  all  members  of  the  command  group. 
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Display  Sensor  Data  interactions  included  255  cursor  moves  over  displayed  icons 
(enemy,  friendly,  and  groups  of  multiple  platforms)  to  access  “Properties”  information. 
Typically,  Properties  information  included  the  type  and  exact  location  of  vehicles.  Command 
group  players  performed  41  interactions  in  support  of  Recognize  Targets,  or  Human  Target 
Recognition  (HTR),  that  involved  display  and  refinement  of  images,  and  updating  icons  to 
reflect  target  type.  Similarly,  the  players  performed  1 87  interactions  in  support  of  Assess  Battle 
Damage,  or  Battle  Damage  Assessment  (BDA),  that  also  involved  the  display  and  refinement  of 
images,  and  updating  icons  to  reflect  target  damage. 

Strike  Function.  Results  reported  in  Table  12  show  that  the  Strike  fimction  was 
accomplished  through  the  performance  of  204  of  the  1,043  total  interactions  (19.5%).  Strike 
related  interactions  were  performed  almost  exclusively  by  the  Battlespace  Manager  and  Effects 
Manager.  Target  Designation  accounted  for  70  interactions:  41  by  the  Battlespace  Manager,  28 
by  the  Effects  Manager,  and  1  by  the  Information  Manager.  Fire  Weapon  actions  included  79 
interactions:  46  by  the  Battlespace  Manager,  32  by  the  Effects  Manager,  and  1  by  the 
Information  Manager.  Seventeen  (17)  interactions  were  performed  to  Monitor  Fires  for  LAM 
missile  engagements  by  the  Effects  Manager,  which  involved  moving  the  cursor  over  the  LAM 
icon  to  bring  up  the  Properties  window.  Monitor  Fires  for  PAM  missile  engagements  included 
seven  interactions  by  the  Battlespace,  Information,  and  Effects  Managers. 

Other.  The  “Other”  category  was  included  to  capture  unanticipated  and  off-task 
activities.  The  only  HCI  task  documented  in  this  category  was  Reboot  System,  which  involved 
selecting  the  reboot  option  and  waiting  for  the  system  to  reboot  after  a  crash.  Only  two  instances 
of  Reboot  System  occurred  during  Run  10  execution,  both  by  the  Battlespace  Manager. 

Frequency  of  HCI  by  Function  and  Time.  It  may  be  useful  to  identify  whether  patterns  of 
interaction  changed  across  different  portions  of  run  execution.  Figure  19  provides  a  comparison 
of  the  frequency  of  interactions  performed  across  Run  10  by  Function  and  Time  (time  quartiles  = 
21  minutes). 
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Figure  19.  Frequency  of  HCI  by  Function  and  Time  Quartile,  Run  10  Execution. 
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Move  related  tasks  represented  5.3%  of  the  interactions  during  Run  10,  with  very  few 
interactions  during  the  first  two  quartiles  prior  to  the  launch  of  the  ground  forces.  The  majority 
of  tasks  performed  (74.4%)  were  See  related  interactions  that  ranged  from  a  high  of  226  during 
the  first  quartile,  to  a  low  of  171  during  the  third  quartile.  Finally,  19.6%  of  the  interactions 
performed  represented  Strike  interactions.  Strike  interactions  showed  a  slight  decrease  in  the 
second  quartile  after  planned  and  AGM  fires  had  taken  place,  followed  by  a  rise  during  the  third 
and  fourth  quartile  as  ground  forces  advanced  along  an  avenue  of  attack. 


Task  Allocation.  Figure  20  summarizes  how  HCIs  were  performed  at  each  screen  and  at 
each  duty  position  during  Run  10  execution.  Notably,  the  two  screens  at  each  workstation  are 
equivalent  and  redundant.  Each  command  group  player  can  choose  which  information  windows 
to  display  and  which  tasks  to  perform  on  each  screen.  Overall,  the  number  of  interactions 
performed  ranged  from  a  high  of  268  on  the  Battlespace  Manager’s  right  screen,  to  a  low  of  39 
on  the  Battlespace  Manager’s  left  screen.  The  Battlespace  Manager  and  Effects  Manager 
performed  most  Strike  related  interactions,  while  the  Cell  Commander  and  Information  Manager 
primarily  performed  See  related  interactions  under  Sensor  Data  Display,  HTR  and  BDA. 


Figure  20  also  indicates  player’s  screen  preference  for  task  performance.  Clear  screen 
preferences  were  the  left  screen  by  the  Effects  Manager,  and  the  right  screen  by  the  Battlespace 
Manager.  The  Commander  and  Information  Manager  used  both  screens  with  approximately 
equal  frequency  to  perform  tasks.  Fimctions  were  not  exclusively  allocated  to  screens.  For 
example,  the  Battlespace  and  Effects  Managers  performed  Strike  related  tasks  on  both  screens. 
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Figure  20.  Frequency  of  HCI  on  left  and  right  screens  by  Fxmction  and  Duty  Position,  Run  10 
Execution. 


Duty  Position  Workload.  In  estimating  workload,  task  frequency,  duration  and  errors 
should  all  be  considered.  Given  the  low  rate  of  time-consuming  tasks  and  performance  errors, 
task  frequency  was  used  as  the  primary  criteria  for  estimating  HCI  duty  position  workload. 
Figure  21  illustrates  the  trend  of  HCI  performance  for  each  player  during  Run  10  execution  by 
presenting  the  frequency  of  task  performance  in  successive  10-minute  intervals,  or  blocks  of 
execution  time.  Note,  the  final  time  segment  (80-84  minutes)  was  deleted  from  Figure  21  to 
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avoid  misleading  comparison  across  different  time  intervals.  The  data  provided  in  Figure  21 
represents  an  emerging  empirical  basis  for  task  allocation  among  the  command  group,  and 
between  the  command  group  and  future  systems. 

A  narrative  description  of  the  command  group  player’s  HCI  tasks  performed  at  each  10- 
minute  interval  of  Run  10  was  developed  to  further  document  how  tasks  vary  over  time  during 
run  execution  (see  Appendix  G).  A  narrative  summary  of  the  information  available  in  Appendix 
G  is  provided  below: 

■  Cell  Commander.  Commander’s  interactions  were  almost  exclusively  (96.2%)  See  related 
interactions.  The  Commander  performed  105  interactions  during  Run  10  execution,  with  the 
greatest  number  occurring  during  the  0-10  and  60-70  minute  time  intervals.  During  these 
intervals,  the  Commander  was  heavily  involved  in  Display  Sensor  Data  tasks  that  involved 
cursoring  over  both  enemy  and  friendly  icons  to  read  properties  information.  Commander’s 
lowest  HCI  workload  occurred  during  the  30-40  minute  time  interval  in  which  he  performed 
no  interactions. 

■  Battlespace  Manager.  The  Battlespace  Manager  performed  (309)  interactions  distributed 
across  Move  (9.0%),  See  (60.3%),  and  Strike  (29.0%)  functions.  Greatest  workload  occurred 
early  in  Run  10  execution  when  he  was  heavily  occupied  in  Display  Sensor  Data  tasks,  and 
Manipulate  Map  tasks  (e.g..  Zoom  and  Scroll).  Workload  appeared  to  decline  to  its  lowest 
level  in  the  40-50  minute  block  during  which  he  fired  the  LOS  weapon  system. 

■  Information  Manager.  The  Information  Manager  performed  more  interactions  (3  3 8)  than 
any  other  command  group  player.  Interactions  included  both  See  (89.7%),  and  Move  (8.0%) 
Functions.  Frequency  of  task  actions  peaked  during  the  40-50  and  70-80  minute  time 
intervals  corresponding  to  high  levels  of  Display  Sensor  Data  tasks,  such  as  cursoring  over 
enemy/friendly  icons  to  read  properties,  and  BDA  tasks.  Workload  dropped  late  in  the  Rim, 
during  the  50-60  minute  time  interval  when  the  Information  Manager  performed  a  lesser 
number  of  Display  Sensor  Data  and  BDA  tasks,  as  well  as  some  Move  Air  Assets  tasks. 

■  Effects  Manager.  The  Effects  Manager  performed  291  interactions  supporting  both  See 
(64.6%)  and  Strike  (35.4%)  Functions.  Highest  workload  occurred  in  (a)  the  10-20 
minute  time  segment  of  the  run,  predominantly  Manipulate  Map  interactions,  and  (b)  during 
the  70-80  minute  time  interval  while  performing  Display  Sensor  Data  interactions,  firing  the 
Netfires  system,  and  Selecting/Changing  Window  display  areas.  Workload  reached  its 
lowest  point  during  the  40-50  minute  time  interval  during  which  he  performed  a  low  number 
of  Display  Sensor  Data  tasks  that  involved  cursoring  over  map  icons  of  vehicles  to  access 
information  on  both  friendly  and  enemy  assets. 
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Figure  21 .  Duty  Position  HCI  load  by  Time,  Run  1 0  Execution. 

HTR  and  BDA  Workload  and  Task  Allocation.  Recognize  Targets  and  Assess  Battle 
Damage,  more  commonly  referred  to  as  HTR  and  BDA,  were  singled  out  for  assessment. 

During  Experiment  2,  tasks  supporting  HTR  and  BDA  were  performed  frequently,  were  time 
consuming,  and  showed  evidence  of  collective  (and  possibly  redundant)  performance. 

Collective  HTR  and  BDA  performance  refers  to  the  situation  where  two  or  more  command 
group  players  examine  the  same  target  image  at  approximately  the  same  time  in  an  effort  to 
reach  consensus,  such  as  in  identifying  the  type  of  target,  or  assessing  target  battle  damage. 
Multiple  reviews  of  the  same  target  image  by  the  same,  or  different,  members  of  the  command 
group  might  also  indicate  redundant  performance.  The  HTR  and  BDA  tasks  required  clicking  on 
a  picture  icon  on  the  map  display,  or  choosing  a  picture  from  a  menu  list  to  bring  it  up  on  a 
display  for  review.  Once  brought  up  the  operator  used  image  location  and  resolution  controls  to 
select  a  portion  of  the  picture  and  adjust  contrast  to  enhance  the  image  for  review. 

During  Experiment  3’s  execution  phase,  109  target  images  were  viewed/reviewed  across 
the  four  workstations.  Six  of  these  images  were  reviewed  for  HTR  (5.5%),  and  103  images 
reviewed  for  the  purpose  of  BDA  (94.5%).  Target  images  were  also  shared  on  the  overhead 
HUD  screen  on  1 1  occasions.  Imagery  analysis  was  relatively  time  consuming  compared  to 
most  HCI  tasks,  requiring  an  average  of  1 7.0  seconds  for  HTR,  and  1 8.2  seconds  for  BDA 
assessment.  Figure  22  presents  the  frequency  of  sensor  image  review  at  each  workstation  for 
purposes  of  HTR  and  BDA.  The  Information  Manager  was  the  primary  image  viewer, 
performing  72  HTR/BDA  tasks.  The  other  command  group  players  performed  fewer  HTR/BDA 
imagery  review  tasks:  Battlespace  Manager  24  images.  Commander  9  images,  and  the  Effects 
Manager  4  images. 
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Figure  22.  Number  of  sensor  images  reviewed  by  Duty  Position,  Run  10  Execution. 

On  occasion,  the  same  sensor  image  was  reviewed  at  more  than  one  workstation  while 
performing  Display  Target  Images,  suggesting  shared  (or  redvmdant)  task  performance.  Of  the 
total  of  109  images  reviewed,  17  (15.6%)  were  reviewed  at  more  than  one  workstation.  The 
Battlespace  and  Information  Manager  reviewed  10  of  the  same  images,  the  Commander  and 
Information  Manager  reviewed  3  of  the  same  images,  the  Effects  and  Information  Manager 
shared  in  the  review  of  3  images,  and  the  Commander,  Battlespace  Manager,  and  Effects 
Manager  all  shared  in  the  review  of  one  sensor  image.  On  eleven  occasions,  command  group 
players  collectively  reviewed  target  imagery  presented  on  the  HUD,  as  noted  previously. 

Notably,  multiple  reviews  of  same  images  during  Experiment  3  were  substantially  less 
than  during  Experiment  2,  respectively  15.6%  versus  61.6%.  Credit  for  this  workload  reduction 
goes  in  large  to  design  changes  in  the  C^  prototype  to  provide  an  audit  trail  on  images  reviewed 
and  by  whom.  This  is  the  type  of  information  processing  assistance  readily  performed  by 
computers  to  improve  the  efficiency  and  effectiveness  of  command  group  performance, 
particularly  for  human  intensive  tasks  such  as  HTR  and  BDA. 

Human-Computer  Interactions — Planning  Phase 

Analysis  of  HCI  during  the  planning  phase  was  introduced  for  Experiment  3,  Run  10. 

The  planning  phase  typically  occurs  immediately  prior  to  the  execution  phase,  and  lasts 
approximately  one  hour.  The  ARFs  decision  to  examine  HCI  during  planning  was  based  on  two 
primary  considerations.  First,  the  evolving  concept  of  FCS  command  and  control  stresses  the 
requirement  for  more  effective  and  collaborative  planning  methods  before  and  during  execution. 
By  design,  the  planning  process  and  products  prior  to  execution  provide  the  basis  for  planning 
changes  during  execution,  sometimes  referred  to  as  dynamic  re-planning.  Second,  planning  for  a 
manned  and  robotic  force,  such  as  the  Unit  Cell,  requires  substantial  changes  in  the  planning 
process  and  products  required  for  execution. 

A  small  command  group  with  robotic  elements  becomes  “doers”  as  well  as  thinkers.  In 
essence,  the  command  group  must  reformulate  battle  commands  into  computer  commands. 
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More  succinct  verbalizations — such  as  conunander’s  intent  and  guidance — ^with  many  implied 
tasks,  must  be  issued  in  computer-mediated/dictated  formats  in  w^hich  many  tasks  must  be 
explicitly  and  precisely  defined.  Directing  and  controlling  robotic  elements  through  human- 
computer  interaction  entails  many  new  tasks  and  many  of  the  new  prototype  features  used  by 

the  participant  command  group  in  Experiment  3. 

Examination  of  new  features  and  tasks  during  planning  is  particularly  important.  For  as 
new  features  become  increasingly  automated  or  “hands-free”  during  execution,  many  of  the  new 
tasks  associated  with  these  features  are  performed  primarily,  sometimes  only,  during  planning. 
By  Experiment  3,  the  prototype  used  by  the  command  group  included  an  array  of  highly 
automated  features  that  supported  each  of  the  basic  functions  xmder  investigation.  A  few 

examples  are  noted  here,  many  more  are  provided  in  the  results  that  follow.  For  the  Plan 
function,  a  Rehearse  Plan  tool  allowed  the  Commander  and  other  members  of  the  command 
group  to  review  a  quick-time  animation  of  their  planned  movements  by  groxmd  and  air  assets. 

For  See,  the  Information  Manager  or  others  could  establish  Auto-Reconnaissance  routes  and 
parameters.  For  Move,  the  Battlespace  Manager  or  others  could  use  a  Create  Routes  function 
that  automatically  generated  route  waypoints  based  on  mobility  and  intervisibility  calculations. 
For  Strike,  the  Effects  Manager  or  others  could  set  the  Attack  Guidance  Matrix  to  automatically 
fire  at  high  priority  targets  during  execution. 

Limitations  to  the  HCI  analysis  of  planning  that  impact  the  results  are  noted.  Some  of 
these  limitations  also  affected  the  verbal  analysis,  as  discussed.  Planning  communications  and 
activities  by  the  command  group  were  often  limited  in  latter  runs,  including  Run  10,  as  the 
command  group  had  performed  very  related  planning  on  prior  runs.  For  example,  planning  with 
respect  to  Terrain  was  curtailed  by  the  command  group’s  prior  experience  with  the  NTC  terrain 
database  during  30+  experimental  runs  (Experiments  1-3),  and  the  players’  live  rotations  at  NTC. 
Therefore,  the  command  group  routinely  re-used  many  of  the  same  graphic  control  measures 
(GCMs)  in  planning  that  they  had  used  during  earlier  runs.  More  specifically,  the  following 
GCMs  were  already  input  in  their  prototypes  at  the  beginning  of  Run  10  planning:  7  attack 
routes,  5  phase  lines,  4  named  areas  of  interest,  12  friendly  vehicles  and  infantry  dismounts,  and 
10  Unattended  Groimd  Sensors  (UGS).  Notably,  the  C^  prototype’s  ability  to  store  prior  plans, 
including  GCMs  and  player-selected  settings  for  automated  features  such  as  AGM,  is  in  general 
a  very  useful  capability. 

In  addition,  the  Information  Manager  assigned  for  Runs  6-1 1  had  replaced  the  original 
Information  Manager,  and  the  Effects  Manager  was  absent  for  approximately  30  minutes  of  the 
planning  phase.  Estimating  the  impact  of  such  personnel  disruptions  is  difficult.  However,  the 
Commander  appeared  to  accept  and  perform  some  of  the  tasks  typically  performed  by  the 
original  and  more  experienced  Information  Manager.  Finally,  all  results  and  conclusions  are 
limited  by  ARI’s  HCI  analysis  of  only  one  experimental  planning  phase. 

The  primary  measures  used  for  assessing  HCI  performance  were  task  duration  and 
frequency,  as  for  execution.  Overall,  the  frequency  as  well  as  the  type  of  HCI  tasks  performed 
differed  across  the  command  group,  as  expected.  Task  frequency  and  type  also  differed  within 
duty  position  in  a  comparison  of  tasks  performed  on  left  versus  right  display  screens  by  each 
player.  Time  and  error  measures  were  less  useful  as  performance  criteria.  Performance  times 
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for  most  HCI  planning  tasks  were  typically  5  seconds  or  less  in  duration,  and  relatively  few 
“errors”  in  HCI  performance  were  detected.  The  only  tasks  associated  with  errors  were 
inadvertent  clicks  on  GCMs,  two  instances  of  accidentally  dropping  an  icon  back  to  its  original 
menu  location,  and  one  example  of  placing  an  incorrect  unit  icon  on  the  map  (sqiiad  instead  of 
platoon). 

Task  Duration.  Time  duration  for  task  completion  was  examined  to  assess  HCI  task 
performance  and  identify  potential  training  requirements.  Tasks  requiring  longer  performance 
times  could  be  candidates  for  redesign,  machine  aiding,  or  additional  training.  Task  performance 
time  was  estimated  by  identifying  Start  and  Stop  actions,  typically  involving  the  selection  of  a 
menu  option,  or  opening  and  closing  a  window.  Overall,  during  Run  10  planning  the  command 
group  players  performed  499  individual  interactions,  less  than  50%  of  thel,043  interactions 
recorded  during  execution.  Approximately  3 1%  of  these  planning  interactions  (153  of  the  499) 
required  more  than  5  seconds  to  perform.  Frequency  data  and  mean  performance  times  for  the 
ten  most  frequently  performed  long  duration  tasks  (133  of  the  153)  are  presented  in  Table  13. 

Notably,  the  same  task  types  listed  in  Table  13  were  sometimes  more  quickly  performed, 
and  required  less  than  5  seconds.  For  example,  the  time  required  to  drag/drop  icons  varied 
considerably.  Accordingly,  Table  13  reports  only  the  26  occasions  in  which  the  Conmander 
took  longer  than  5  seconds  to  drag/drop  icons.  In  fact,  the  Commander  performed  this  task  70 
times  during  the  planning  session,  but  on  the  other  44  occasions  the  Commander  took  less  than  5 
seconds  to  ^ag/drop  an  icon. 

Table  13 

HCI  Long*  Duration  Tasks  by  Frequency  and  Time,  Run  10  Planning 


Function 

Task 

Frequency  and  Task 

Time  in 
Seconds 
(Mean/SD) 

Commander 

Battlespace 

Information 

Effects 

Plan 

Place  Platforms  on  Map 

1 

— 

9 

— 

16.7/12.3 

Rehearse  the  Plan 

— 

6 

7 

— 

28.8/13.4 

Move  Icons  (Drag/Drop) 

26 

3 

13 

— 

14.7/13.0 

Modify  Overlay  Graphics 

1 

— 

7 

— 

21.4/19.8 

Move 

Create  Ground  Routes 

— 

6 

— 

— 

22.0/8.6 

Create  Air  Routes 

— 

— 

21 

— 

21.1/11.1 

See 

Plot  Intervisibility 

14 

— 

— 

22.6/13.5 

Select/Change  Windows 

1 

4 

— 

20.8/13.0 

Change  Icon  Type 

1 

— 

6 

2 

18.0/12.0 

Strike 

Fire  PAM 

— 

— 

— 

5 

13.2/4.7 

*Long  duration  tasks  require  more  than  five  seconds  to  perform. 
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Frequency  by  Function  and  Duty  Position.  Figure  23  provides  a  summary  overview 
of  the  HCI  tasks  performed  by  Function  during  Run  10  planning.  Of  the  499  total  interactions 
performed  during  planning,  their  frequency  by  function  was:  Plan  (179),  See  (256),  Move  (52), 
and  Strike  (12).  As  for  execution,  the  greatest  number  of  interactions  performed  during  planning 
supported  the  command  group’s  ability  to  See  or  visualize  the  battlefield,  particularly  the 
weapon  system  identities  and  characteristics  of  friendly  and  templated  enemy  assets.  Table  14 
presents  an  over\dew  summary  of  the  task  frequency  data  for  the  9  of  17  Sub-Functions  and 
the  26  of  83  HCI  tasks  available  that  were  actually  performed  during  Run  10  planning.  A  brief 
narrative  summary  of  these  results  by  Function  is  provided  below. 


Function 


Figure  23.  Frequency  of  HCI  by  C^  Function,  Run  10  Planning. 
Table  14 


Frequency  of  C^  Sub-Functions  and  HCI  by  Duty  Position,  Run  10  Planning 


Function 

Subfunction 

Task 

Commander 

Freq.  % 

Battlespace 

Manager 

Freq.  % 

Information 

Manager 

Freq.  % 

Effects 

Manager 

Freq.  % 

Plan 

76 

63.3 

42 

25.8 

mm 

35.7 

1 

2.1 

Create  Mission/COA 

76 

63.3 

42 

25.8 

35.7 

1 

2.1 

Create  Overlay  Graphics 

— 

— 

1 

.6 

— 

— 

— 

— 

Place  Platforms  on  Map 

1 

38 

— 

— 

14 

8.3 

— 

— 

Rehearse  the  Plan 

4 

3.4 

10 

6.1 

8 

4.8 

— 

— 

Move  Icons  (Drag/Drop) 

58.3 

21 

12.9 

28 

1.7 

1 

2.1 

Modify  Overlay  Graphics 

1 

.8 

10 

6.1 

— 

— 

Move 

— 

— 

25 

15.3 

26 

15.5 

1 

2.1 

Move  Ground  Assets 

— 

— 

24 

14.7 

5 

1 

2.1 

Create  Routes 

— 

— 

7 

4.3 

— 

— 

— 

— 

Edit  Existing  Route 

— 

— 

4 

2.5 

— 

— 

— 

— 

Delete  all  Tasks 

— 

— 

8 

4.9 

5 

3.0 

1 

2.1 

Create  Overwatch  Task 

— 

— 

1 

.6 

— 

— 

— 

— 

Table  Continues 


43 


Generate  Route 

Move  Air  Assets 

Recon  an  Area 

Group  Follow 

Create  Ground  Follow 

— 

— 

4 

1 

1 

2.5 

.6 

.6 

21 

21 

12.5 

12.5 

— 

— 

See 

44 

36.7 

96 

58.9 

82 

48.8 

34 

Manipulate  M^ 

10 

8.3 

10 

6.2 

15 

8.9 

18 

37.5 

Zoom  Map 

8 

6.7 

6 

3.7 

12 

7.1 

7 

14.6 

Scroll  Map 

2 

1.6 

4 

2.5 

3 

1.8 

11 

22.9 

Use  Visualization  Aids 

6 

47 

28.8 

14 

83 

2 

4.2 

Plot  Intervisibility 

— 

— 

27 

16.6 

— 

— 

— 

— 

Display  on  Heads  Up 

— 

— 

2 

1.2 

— 

— 

— 

— 

Select/Change  Windows 

6 

5.0 

11 

6.7 

14 

8.3 

2 

4.2 

Change  GCM  Settings 

— 

— 

7 

4.3 

— 

— 

— 

— 

Display  Sensor  Data 

26 

21.7 

39 

23.9 

44 

26.2 

12 

^1 

Query  Enemy 

17 

14.2 

10 

6.1 

31 

18.5 

7 

14.6 

Query  Friendly 

9 

7.5 

27 

16.6 

13 

7.7 

5 

mm 

Toggle  Sensor  Fans 

;  — 

— 

2 

1.2 

— 

— 

— 

— 

Recognize  Targets 

2 

1.7 

— 

— 

9 

5.4 

2 

4.2 

Change  Icon  Type 

2 

1.7 

— 

— 

9 

5.4 

2 

4.2 

Strike 

— - 

— 

— 

— 

12 

25.0 

Fire  a  Weapon 

— 

— 

— 

— 

— 

— 

12 

25.0 

Fire  LAM 

— 

— 

— 

— 

— 

— 

3 

6.3 

Fire  PAM 

— 

— 

— 

— 

— 

— 

6 

12.5 

Fire  LOS 

— 

— 

— 

— 

— 

— 

1 

2.1 

Delete  Fire  Tasks 

— 

— 

— 

— 

— 

— 

2 

4.2 

Total 

120 

100 

1  163 

100 

168 

100 
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100 

■  Plan  Function.  For  Run  10  planning,  179  of  the  499  total  interactions  (35.87%)  were 
performed  to  support  the  Plan  Function.  These  interactions  primarily  involved  moving 
existing  vehicle  icons  and  graphic  control  measure  lines  (120  interactions)  into  their  initial 
starting  positions  on  the  tactical  map  using  the  drag/drop  icons  interface  capability.  The 
Rehearse  Plan  tool  was  used  by  the  Commander,  Battlespace,  and  Information  Managers  a 
total  of  22  times  to  observe  and  compare  the  planned  movement  of  both  ground  and  air 
vehicles,  and  dismounted  Land  Warrior  teams.  In  particular  the  Rehearse  Plan  tool  was  used 
by  the  Battlespace  Manager  to  brief  the  Commander  regarding  the  planned  movement  of 
ground  assets. 

■  See  Function.  The  majority  (5 1 .0%)  of  Run  10  planning  interactions  supported  the  See 
function.  The  most  frequent  See  activities  involved  using  the  Query  Enemy/Friendly  tool, 
cursoring  over  vehicle  icons  to  bring  up  Enemy  (65  interactions),  and  Friendly  (54 
interactions)  asset  properties  information  such  as  the  type,  and  exact  position  of  vehicles. 

The  use  of  the  Plot  Intervdsibility  tool  was  also  a  frequently  used  feature  (27  interactions)  in 
planning  ground  asset  movement  routes. 

■  Move  Function.  The  Move  function  was  accomplished  through  the  performance  of  52  of  the 
499  total  (10.6%)  interactions.  These  Move  related  interactions  involved  creating  future 
ground  and  air  vehicle  movement  routes,  typically  by  entering  route  points  and  by  clicking 
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locations  on  the  map.  The  Battlespace  Manager  performed  25  interactions  associated  with 
creating  routes  for  ground  vehicles,  and  the  Information  Manager  performed  21  interactions 
to  create  routes  for  air  sensor  assets. 

■  Strike  Function.  Only  12  interactions  recorded  for  Run  10  planning  supported  the  Strike 
function,  all  performed  by  the  Effects  Manager.  The  relatively  low  number  of  Strike-related 
interactions  during  planning  may  reflect  the  introduction  of  the  AGM  that  provided 
automated  fires  against  a  number  of  operator-defined  target  contingencies.  As  no  AGM  tasks 
were  recorded  during  the  planning  phase  for  Run  10,  the  AGM  settings  from  Run  9  were 
probably  used  to  begin  the  execution  phase  of  Run  10.  As  noted,  the  low  number  of  HCI 
interactions  related  to  Strike  may  have  resulted  from  the  Effects  Manager  being  absent  from 
the  vehicle  for  approximately  30  minutes  during  Rim  10  planning. 

Figure  24  provides  a  summary  overview  of  the  interactions  performed  by  duty  position 
during  Run  10  plaiming.  In  order  by  amomt,  the  percentage  of  See  related  interactions 
performed  by  each  player  was  Effects  Manager  (70.8%  of  his  total  tasks),  Battlespace  (58.9%), 
Information  Manager  (48.8%),  and  Commander  (36.7%).  Interactions  supporting  the  Plan 
function  were  the  next  most  frequently  performed.  Of  the  total  interactions  performed  by  each 
player,  the  percentage  of  Plan  related  interactions  by  position  were:  Commander  (63.3%), 
Information  Manager  (35.7%),  Battlespace  Manager  (25.8%),  and  Effects  Manager  (2.1%). 

Move  related  interactions  were  less  frequent  and  performed  primarily  by  the  Information 
Manager  and  the  Battlespace  Manager,  respectively  15.5%  and  15.3%  of  their  total  interactions. 
Only  12  instances  of  Strike  related  HCI  tasks  were  recorded,  all  performed  by  the  Effects 
Manager,  representing  25%  of  his  total  interactions  during  planning. 


Duty  Position 

Figure  24.  Frequency  of  HCI  by  Duty  Position,  Run  10  Planning. 

The  most  frequent  interactions  performed  by  the  Commander  were  to  Create  Mission/ 
Course  of  Action  (COA)  (76  interactions),  and  Display  Sensor  Data  (26  interactions).  The 
Commander  performed  Create  Mission/COA  tasks  primarily  to  move  existing  vehicle  and  GCMs 
on  the  map  display,  to  modify  existing  graphics,  and  to  rehearse  the  plan.  The  Commander 
performed  Display  Sensor  Data  tasks  primarily  to  access  the  properties  of  enemy  icons.  The 
most  frequent  interactions  performed  by  the  Battlespace  Manager  were  Use  Visualization  Aids 
(47  interactions),  Create/Update  Course  of  Action  (COA)  (42  interactions),  and  Display  Sensor 
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Data  (39  interactions).  The  most  frequent  interactions  for  the  Information  Manager  were  Create 
Mission/COA  (60  interactions).  Display  Sensor  Data  (44  interactions),  and  Move  Air  Assets  (21 
interactions)  by  creating  routes.  The  most  frequent  interactions  for  the  Effects  Manager  were 
Manipulate  Map  (18  interactions)  that  primarily  involved  Zoom  and  Scroll  tools.  Display  Sensor 
Data  (12  interactions),  and  the  use  of  the  Fire  Weapon  tool  (12  interactions)  to  plan  fires  for  the 
PAM  and  LAM  missiles,  and  the  LOS  gun. 

Task  Allocation  and  Workload.  Task  allocation  and  workload  during  the  planning  phase 
by  FCS-type  small  command  groups  are  important  and  largely  neglected  issues.  These  issues  are 
at  best  only  indirectly  addressed  by  the  results  provided  in  this  section.  The  ARI’s  prior  caveats 
about  the  exploratory  nature  of  this  report’s  planning  results  apply  especially  here.  Brief 
narrative  descriptions  provided  below  highlight  task  allocation  and  workload  issues.  These 
descriptions  provide  a  coherent  summary  of  key  interactions  performed  by  each  command  group 
player  during  Run  1 0  planning. 

■  Commander.  The  C^  prototype  provided  a  shared  view  of  the  planning  process  and  products 
(such  as  icon  emplacement,  route  generation  and  terrain  consideration)  that  enabled  the 
Commander  to  monitor  the  inputs  of  other  players  and  correct  mistakes  early.  For  example, 
“You’ve  got  the  wrong  unit  there,  it  should  be  a  platoon  size  element.”  The  C  prototype 
provided  redundant  capabilities  across  duty  positions.  This  allowed  the  Commander  to 
emplace  enemy  templates  and  unmanned  ground  sensors,  so  the  Information  Manager  could 
focus  on  creating  and  updating  named  areas  of  interests  (NAIs).  Drag/drop  placement  of 
UGS  was  a  time  consuming  task.  The  Commander  first  placed  UGS  in  general  areas,  then 
zoomed  to  high  magnification  to  adjust  placements  based  on  his  best  estimate  of  enemy 
routes.  In  particular,  the  Commander  provided  over-the-shoulder  guidance  to  players,  and 
used  the  Rehearsal  tool  to  discuss  the  plan  with  other  players. 

■  Battlespace  Manager.  The  Battlespace  Manager  used  the  HUD  to  present  his  plan  of 
movement  to  the  Commander:  “So  this  is  my  route  plan.”  Presentation  included  showing 
vehicle  intervisibility  as  he  moved  his  cursor  along  the  planned  movement  route,  identifying 
a  templated  tank  that  had  to  be  engaged,  and  using  the  Plot  Intervisibility  tool  to  show 
unrestricted  visibility  across  the  valley.  This  presentation  included  verbal  brief  of  the  route 
plan  to  the  Conunander.  He  used  the  HUD  to  demonstrate  sensor  coverage  for  the 
Commander  by  using  the  Toggle  Range  Fans  tool  while  explaining:  “This  is  direct  vision 
optics  (DVO)  and  infra  red  (IR).  I  can  turn  the  Fire  Finder  fan  on.  We  are  going  to  be  able 
to  see  out  to  here.”  By  using  the  HUD,  the  Effects  Manager  was  able  to  comment  on  the 
Battlespace  Manager’s  plan  by  identifying  the  location  of  planned  PAMs  and  LAMS  against 
targets.  The  Battlespace  Manager  used  the  Rehearse  Plan  tool  to  create  and  refine  automated 
routes,  confirming  that  vehicles  would  move  as  desired  in  the  group  follow  mode,  and 
working  with  the  Commander  to  identify  needed  adjustments. 

■  Information  Manager.  The  Information  Manager  performed  numerous  drag/drop  placements 
of  enemy  icon  templates  and  numerous,  mainly  fruitless,  queries  on  enemy  templates.  The 
Information  Manager  often  used  the  Rehearsal  tool  after  creating  movement  routes  for  the 
Shadow  and  the  four  Micro-UAVs  to  confirm  desired  routes  had  been  entered.  In  particular, 
air  routing  and  imagery  analysis  tasks  contributed  to  his  workload.  As  stated  to  the 
Commander:  “I  have  two  Micro-UAVs  taking  pictures  of  every  one  of  those  templated 
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targets  out  there.  Then  I  will  have  one  for  NAI 1,  and  one  for  NAI 2.  Can  have  Micro¬ 
ti  AV2  or  Micro-UAV3  looking  at  the  routes.” 

■  Effects  Manager.  Prior  to  his  absence,  the  Effects  Manager  created  some  firing  tasks, 
observable  as  missile  fly-outs  during  rehearsals  run  by  the  command  group,  particularly  the 
Commander.  The  AGM  appeared  to  shift  the  allocation  of  some  Strike  task  decisions  to 
automation,  particularly  for  high  priority  targets,  and  reduce  the  overall  human  requirement 
to  plan  and  set  fires.  As  noted,  the  Effects  Manager  did  not  revise  the  AGM  during  Run  10 
planning.  In  addition,  automated  features  such  as  AGM  may  help  reduce  personnel 
requirements,  at  least  temporarily.  When  faced  with  starting  Run  10  execution  without  the 
Effects  Manager,  the  Commander  commented  that  he  could  assume  the  Effects  Manager 
tasks  temporarily:  “Well,  initially  with  the  AGM  and  everything,  I  could  do  the  quick  fires.” 

Table  15  provides  a  bulleted  summary  of  the  same  HCI  tasks  performed  by  each 
command  group  player  during  Run  10  planning. 

Training  Implications.  Human  performance  during  the  Planning  phase  needs  to  be  more 
closely  and  rigorously  examined.  At  best,  planning  results  should  provide  valuable  insights  into 
training  requirements  based  on  task  identification  and  task  allocation.  Notably,  many  key  and 
time  consuming  tasks — such  as  creating  and  verifying  vehicle  air  and  groimd  routes,  planning 
and  verifying  automated  fires,  and  intelligence  planning  and  preparation — ^are  performed  during 
planning.  Training  implications  are  discussed  in  more  detail  in  the  Conclusions  section. 

Table  15 

Key  Planning  Functions  and  Tasks  by  Duty  Position,  Rim  10  Plaiming 


Cell  Commander 

Battlespace  Manager 

Information  Manager 

Effects 

Manager 

See  and  understand 

Plan  ground  sensor  routes. 

Develop  Intelligence 

Plan  LAM 

the  battlefield. 

overwatch  positions,  and 

collection  plan  with 

and  PAM 

employment  of  two  LOS 

NAIs  to  identify 

fires  for  (2) 

Develop,  prepare,  and 
synchronize  CO  As. 

weapons. 

enemy  COA. 

Create  routes  for 

Netfire 

systems. 

Create  routes  for  C^ 

Create  routes,  group  follow. 

Shadow  UAV  and 

Create  Netfire 

Vehicle. 

and  overwatch  taskings  for: 
Line  of  Sight  Vehicle  (2) 

four  Micro-UAVs. 

ground  routes. 

Emplace  Threat  Icon 

Roboscout  Vehicle  (2) 

Create  area/target 

Create/update 

Templates  on  C^ 

Future  Warrior  Vehicle  (2) 

reconnaissance  tasks 

the  Attack 

map. 

Future  Warrior  Team  (2) 

and  picture  taking 

Guidance 

Javelin  Team  (2) 

tasks  for  UAVs. 

Matrix 

(AGM) 
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Automated  Measures  Validation  and  Refinement 


As  preface,  the  development  and  validation  of  automated  measures  is  an  iterative  process. 
This  process  generally  requires  the  collaborative  efforts  of  behavioral,  technical,  ^d  operational 
subject  matter  experts.  Behavioral  scientists  often  initiate  the  process  by  identifying,  describing 
and  defining  the  measures  of  interest.  Technical  experts  then  attempt  to  develop  the  measures 
specified,  including  the  software  codes  required  to  identify  and  log  the  measures  as  defined.  As 
will  become  clear  in  the  results  below,  there  are  often  discrepancies  between  behavioral  inputs 
and  technical  outcomes.  Resolving  such  discrepancies  requires  additional  refinements,  often  by 
the  behavioral  and  technical  experts.  Ultimately,  operational  experts  must  help  determine  the 
practical  utility  of  the  automated  measures  developed  and  validated. 

The  results  provided  below  are  basically  a  status  report  on  the  PCS  program’s  efforts 
to  develop  and  validate  automated  measures.  Automated  measures  designed  to  automatically 
capture  data  on  command  group  interactions  with  their  C  prototypes  to  support  training, 
evaluation  and  system  design.  Upon  completion  of  the  manual  HCI  analysis  of  Run  1 0, 
manually  tabulated  frequencies  of  command  group-C^  prototype  interaction  were  compared  with 
the  three  automated  measures  developed  for  Experiment  3. 

Overall,  the  results  are  meager  but  promising.  Two  of  the  three  automated  measures 
developed  for  Experiment  3  were  at  least  partially  validated.  Efforts  to  validate  the  measure 
Alert  Acknowledged  were  not  successful.  As  indicated  in  the  list  of  requested  measures  in  Table 
5,  the  intent  of  this  measure  was  to  capture  the  time  to  respond  to  an  alert  by  turning  it  off. 
However,  the  parsed  files  for  Alert  Acknowledged  appeared  to  include  unnecessary  and 
confusing  information.  For  example,  time  data  included  two,  sometimes  three,  different  time 
stamps  that  could  not  be  readily  distinguished  or  matched  to  the  run  times  available  with  the 
video  recordings  used  for  manual  data  reduction.  Images  associated  with  some  alerts  were  ^ 
identified  on  the  log  with  system  file  names  (e.g.  \\ucl-images\images\ir_10.ntf  )  versus  the  C 
prototype  image  names  (e.g.,  Garm  23)  available  with  the  video  recordings.  In  addition,  the  C 
prototype  provides  an  array  of  options  for  setting  and  responding  to  alerts  that  complicates 
measure  definition  and  extraction. 

Efforts  to  validate  the  measures  Images  Requested A^ie wed  and  Create  Ground/Air  Route 
were  fairly  successful.  Table  16  summarizes  the  validation  results  with  a  comparison  of  the 
frequencies  tabulated  for  these  two  measures  by  automated  and  manual  measurement  methods. 
There  were  no  discrepancies  in  the  number  of  images  requested/viewed  by  the  Commander  and 
Effects  Manager  who  accessed  their  images  from  the  Battlefield  Assistant  in  the  Alert  Window. 
However,  there  were  notable  discrepancies  in  this  same  measure  for  the  Battlespace  and 
Information  Manager.  These  discrepancies  were  probably  due  to  shortcomings  in  the  developed 
measure  that  did  not  count  images  viewed  by  clicking  Chaser  round  icons  that  only  appeared  on 
the  Map  Window. 

Similarly,  there  were  no  discrepancies  in  the  measured  number  of  ground  routes  created 
by  the  Battlespace  Manager.  There  was  only  a  small  discrepancy  (19  versus  18)  in  the  measured 
number  of  air  routes  created  by  the  Information  Manager.  This  appeared  to  be  due  to  the  faet 
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that  manual  measures  tabulated  only  the  number  of  air  routes  actually  generated,  rather  than  the 
number  of  air  routes  attempted  but  not  successfully  completed. 

In  sum,  the  results  on  the  development  and  validation  of  automated  measures  are  quite  limited 
but  promising.  Discrepancies  identified  for  Images  RequestedA/^iewed  and  Create  Ground/Air  Route 
should  be  resolved  with  collaboration  by  behavioral  and  technical  experts.  Clearly,  additional 
development,  refinement  and  validation  are  needed  to  develop  a  useful  set  of  automated  measures  to 
support  training,  evaluation  and  C'  system  design. 

Table  16 

Comparison  of  Automated  Measures  to  Manual  HCI  Data  Reduction 


Duty  Position 

Measured  task 

Frequency  Accounted  For 

Automated  Measures  Manual  HCI  Analysis 

Commander 

Images  RequestedA^iewed 

9 

9 

Battlespace  Manager 

Create  Ground  Route 

11 

11 

Images  Requested A^ie wed 

2 

24 

Information  Manager 

Create  Air  Route 

19 

18 

Images  RequestedA^iewed 

62 

72 

Effects 

Images  RequestedA^iewed 

4 

4 

Subjective  Measures 

Results  from  subjective  measures  are  based  on  the  battery  of  measures  developed  and 
administered  by  ARI  during  Experiment  3.  Overall,  the  results  from  10  different  survey  and 
questionnaire  instruments  are  reported.  Most  of  these  results  were  provided  in  the  Rapid 
Reaction  Report  provided  to  the  Program  Manager  (PM)  FCS  C^  in  November  2002.  They  are 
included  here  to  provide  the  PM  consolidated  documentation  on  ARTs  efforts  for  Experiment  3. 
This  section  begins  with  an  examination  of  an  informal  interview  conducted  in  the  C^  vehicle 
immediately  after  each  run  called  the  In-Place  After  Action  Review  (AAR). 

In-Place  After  Action  Review:  In  Their  Own  Words.  The  command  group  player 
obseiv'ations  and  assessment  during  the  In-Place  AARs  provides  an  informative  summaiy'  of 
each  run  from  Experiment  3  in  the  players'  own  words.  These  observations  are  of  particular 
interest  as  they  were  recorded  directly  after  each  nan,  and  reflect  the  command  group's  most 
immediate  assessment  of  the  just  completed  Unit  Cell  operations.  The  In-Place  AAR  documents 
specific  issues  and  observations  associated  with  each  run  from  the  players'  perspective.  A  small 
sample  of  the  observ'ations  provided  by  the  four  players  during  the  In-Place  AAR  follows: 
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Battlespace  Manager  Comments: 

■  Run  8.  I  tried  the  Group  Overwatch  function.  It 'worked  m planning.  It  generated  a  decent 
route  to  move  the  cluster  forward  through  first  terrain  box.  It  didn’t  work  in  execution. 
Robo-scout  GSR  (ground  surveillance  radar)  went  out  alone,  but  others  didn’t  follow.  So 
FIFV  (Future  Infantry  Fighting  Vehicle)  and  dismounts  not  into  the  fight. 

■  Run  10.  We  had  a  good  plan,  and  we  are  executing  it  well.  But  we  can’t  tell  if  a  target  was 
killed  until  the  AAR.  We  wanted  to  attack  the  enemy’s  strength.  I’m  not  confident  in  Group 
Follow  and  terrain  reasoning.  The  wrong  vehicle  was  in  the  lead  on  the  group  follow  task. 
We  had  a  problem  with  the  FIFV  taking  artillery. 

■  Run  11.  Pretty  happy.  Individual  route  generation  worked.  We  synched  assets  using  the 
synchronization  matrix.  We  dismounted  the  infantry  and  bounded  forward. 

Information  Manager  Comments: 

■  Run  8.  Good  mn,  sensors  working.  The  problem  was  clutter.  The  graphics  for  BDA  (battle 
damage  assessment),  Chasers,  Targets,  and  Spot  Reports  are  all  stacked  on  top  of  each  other. 
Message  icon  is  black  and  I  can’t  see  through  it  to  other  information.  Didn’t  have  time  to 
drill  down  through  them.  Tried  to  change  target  states,  but  it  looked  like  AGM  (attack 
guidance  matrix)  auto  fired  when  state  changed. 

■  Run  1 0.  Biggest  thing,  the  UAV’s  (unmanned  aerial  vehicles)  are  going  out  on  their  own. 
The  Shadow  took  off  on  its  own  without  me  knowing  about  it.  I  got  wrapped  up  in  imagery 
and  didn’t  realize  how  long  it  was  taking.  Same  problem  with  micro-UAVs  too,  they  went 
off  on  their  own  and  I  had  to  keep  pulling  them  back. 

■  Run  1 1 .  We  kept  the  Shadow  alive,  couldn’t  task  the  A1 60.  Shadow  discovered  lots  of 
targets.  Very  late  lost  some  micro-UAVs.  Named  areas  of  interest  not  effective. 

Effects  Manager  Comments: 

■  Run  8.  Good  run,  hit  a  lot  of  stuff  Bad  thing  was  that  we  hit  the  same  targets  repeatedly, 
loitering  attack  missile/munitions  (LAMs)  didn’t  do  redirect.  I  need  to  work  circular  error 
probable  (CEP)  when  engaging  closely  located  targets. 

■  Run  10.  Went  fairly  well.  I  tried  to  work  on  the  problem  of  missiles  guiding  in  on  past 
targets.  I  tried  different  search  radiuses,  but  kept  on  having  the  same  problem  of  subsequent 
missiles  guiding  in  on  same  target. 

■  Run  1 1 .  We  shot  a  lot,  46  missiles,  and  got  the  unattended  ground  sensors  (UGS)  into  the 
fight.  We  put  up  a  wall  of  LAMs  based  on  template  of  enemy  air  defense  artillery  (ADA). 
Auto  redirect  of  LAMs  worked  when  targets  popped  up.  We  need  to  watch  this  because  the 
closest  LAM  didn’t  go  to  closest  target. 

Unit  Cell  Commander  Comments: 

■  Rim  8.  Planning  went  well.  Intelligence  was  pretty  good.  We  lost  the  synthetic  aperture 
radar  (SAR)  early,  tasked  the  A 160.  lam  more  concerned  that  (1)  precision  attack 
missiles/munitions  (PAMs)  hit  targets  that  were  already  hit,  and  (2)  BDA  is  still  a  challenge 
for  us,  we  keep  double  tapping  targets.  Will  keep  micro-UAVs  right  in  front  as  we  move. 
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■  Run  10.  Our  plan  to  concentrate  Intelligence  worked.  We  are  working  at  synchronizing 
systems,  and  working  at  developing  tactics,  techniques  and  procedures  (TTP’s).  It  is  real 
hard  to  tell  if  we  are  hitting  targets.  We  would  do  a  double  tap  and  still  see  target  moving. 
BDA  still  missing. 

■  Run  1 1 .  We  had  good  situational  awareness.  We  tried  to  operational  control  (OPCON)  the 
A160.  The  real  wiimer  today  was  moving  target  indicators  (MTI).  We  used  a  tactical  move, 
and  our  lesson  learned  is  to  conduct  overwatch  with  the  GSR.  We  need  to  learn,  practice, 
and  develop  TTPs  on  how  to  OPCON  the  A 160. 

Workload  and  Performance  Success.  One  key  question  for  Experiment  3  was  whether  or 
not  the  addition  of  the  new  features  into  the  prototype  would  alter  player  workload  ratings. 

Figure  25  provides  summary  results  on  perceived  workload  across  all  Experiment  3  runs  by  duty 
position  and  Complexity.  These  workload  ratings  were  calculated  by  averaging  player  ratings 
across  Mental,  Physical,  Temporal,  Effort,  and  Frustration  scales.  Overall,  the  figure  depicts  a 
clear  increase  in  perceived  workload  in  a  comparison  of  Medium  versus  Too  High  levels  of  run 
complexity.  This  increase  is  not  uniform  across  players;  workload  increase  in  the  Too  High 
runs  was  most  pronounced  for  the  Information  and  Battlespace  Managers. 

Figure  26  provides  summary  results  on  performance  success  across  all  Experiment  3  runs 
by  duty  position  and  Complexity.  These  ratings  were  based  on  responses  to  a  question  that 
asked  players  to  rate  “How  successful  were  you  in  accomplishing  what  you  needed  to  do?”  after 
each  run.  All  four  players  reported  a  decline  in  performance  success  in  the  Too  High  level  of 
Complexity.  The  expected  decline  in  performance  success  was  greatest  for  the  Information 
Manager  and  least  for  the  Effects  Manager.  The  Information  Manager’s  lower  estimates  of 
performance  success  may  reflect  difficulties  in  controlling  the  Unit  Cell’s  Shadow  UAV  and  the 
Micro-UAVs,  and  large  amounts  of  target  imagery  analysis  required  in  the  Too  High  condition. 

(f  Prototype  Support  of  Functions  and  METT-TC  Factors.  Figure  27  provides 
summary  results  on  the  C^  prototype’s  effectiveness  for  Plan,  Move,  See,  and  Shoot  functions. 
Average  estimates  of  the  prototype  effectiveness  for  C^  functions  ranged  from  “Neutral”  to 
“Very  Effective.”  Overall,  the  results  indicate  substantial  room  for  improving  the  C^  prototype, 
particularly  for  the  See  function.  Different  estimates  by  duty  position  should  be  noted.  For 
example,  ratings  of  effectiveness  in  support  of  Move  differed  considerably  between  the  Cell 
Commander  (“Neutral”),  Battlespace  and  Information  Managers  (“Effective”),  and  Effects 
Manager  (“Very  Effective”).  Written  comments  cited  difficulties  in  accomplishing  BDA  and 
target  declutter,  and  the  ineffectiveness  of  Micro-UAVs  for  seeing  the  battlefield.  Commander 
recommended  adding  intervisibility  lines,  and  better  mobility/counter-mobility  graphics. 
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Figure  25.  Average  Workload  Ratings  Across  Runs  by  Duty  Position  and  Run  Complexity. 
Scale  Values:  0  =  Very  Low,  50  =  Moderate,  100  =  Very  High 
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Figure  26.  Average  Performance  Success  Ratings  Across  Runs  by  Duty  Position  and  Run 
Complexity. 

Scale  Values:  0  =  Failure,  50  =  Moderate,  100  =  Perfect 

Figure  28  provides  summary  results  on  the  C^  prototype’s  effectiveness  for  Mission, 
Enemy,  Troops,  Terrain,  Time,  and  Civilians  factors.  Average  estimates  of  the  prototype^ 
effectiveness  for  these  factors  ranged  from  “Neutral”  to  “Very  Effective.’  Overall,  the  C 
prototype  was  rated  “Effective”  or  better  on  21  of  24  total  ratings  (85%).  “Neutral”  ratings 
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should  be  attended  to,  however.  Written  comments  cited  difficulties  that  led  to  targeting  and 
engaging  civilian  or  non-hostile  vehicles. 


Plan  Move  See  Shoot 

Function 

Figure  27.  Mean  Ratings  of  Effectiveness  of  Prototype  by  Function  and  Duty  Position. 
Scale  Values:  1  =  Very  Ineffective,  3  =  Neutral,  5  =  Very  Effective;  Run  1 1 
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Figure  28.  Mean  Ratings  of  Effectiveness  of  Prototype  by  METT-TC  Factor  and  Duty 
Position. 

Scale  Values:  1  =  Very  Ineffective,  3  =  Neutral  5  =  Very  Effective;  Run  1 1 


Skill  Proficiency.  Figure  29  provides  summary  results  on  player  ratings  of  skill 
proficiency.  Ratings  of  skill  proficiency  prior  to  the  experimental  runs  were  generally  lower 
than  after-run  ratings,  as  expected.  Collective  technical  skills  for  using  the  prototype  and 
directing  robotic  assets  received  the  lowest  ratings.  Run  experience  clearly  resulted  in  higher 
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ratings  of  skill  proficiency  in  general.  However,  a  pattern  of  increased  proficiency  with 
increased  run  experience  was  not  strongly  supported. 

Two  possible  reasons  why  a  pattern  of  increasing  proficiency  was  not  found  are  briefly 
examined.  First,  substitutions  in  personnel  and  reassigned  duty  positions  across  runs  confound 
individual  skill  and  particularly  collective  skill  proficiency  estimates.  Second,  “sometimes  you 
don’t  know  what  you  don’t  know.”  Perhaps,  only  as  the  players  began  to  use  and  struggle  with 
the  new  prototype  features  did  they  begin  to  realize  the  intricacies,  complexities,  and 
unexpected  consequences  of  this  automation. 

At  times,  automation  failed  to  perform  as  expected.  At  other  times,  it  performed  in 
unexpected  and  detrimental  ways.  Even  during  the  latter  runs  of  Experiment  3,  the  causes  of  the 
automation  problems  being  experienced  were  often  unclear.  Was  it  a  shortcoming  in  the 
technology,  training,  or  TTPs?  Often,  no  one  on  the  operational  or  technical  teams  could 
precisely  identify  the  exact  cause  of  the  automation  problems  experienced  during  Experiment  3. 
A  similarly  unclear  pattern  of  results  is  provided  in  the  following  section  on  workload  associated 
with  new  prototype  features. 


o 

Technical  and  Tactical  Skills 

Figure  29.  Command  Group  Player  Proficiency  Ratings  of  Technical  and  Tactical  Skills. 

Scale  Values:  1  =  Extremely  Low,  5  =  Average,  9  =  Extremely  High 

Workload  on  New  Prototype  Features.  Figure  30  provides  a  summary  of  the  results 
on  average  workload  across  players  on  selected  new  prototype  features  for  Run  2  and  Run  9. 
For  clarity,  scale  values  w’ere  reversed  so  that  larger  numbers  are  associated  with  higher 
workload.  After  Run  2,  all  the  new  prototype  features  were  rated  as  decreasing  workload,  except 
for  Group  Tasking.  After  Run  9,  however,  the  HTR  Viewer  and  BDA  Recommendation  features 
were  rated  as  increasing  workload.  Similarly,  Run  2  comments  were  all  positive,  while  Run  9 
comments  were  generally  negative.  Again,  changes  in  personnel  and  mn  conditions  may 
contribute  to  the  reported  workload  differences  between  Runs  2  and  Run  9.  However,  some  new 
features  were  consistently  rated  as  work  reducers,  particularly  the  AGM,  New  Alerts  and  Quick 
Fire  features. 
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Figure  30.  New  Prototype  Features  Impact  on  Command  Group  Workload. 
Scale  Values:  1  =  Decreased  Greatly,  3  =  No  Change,  5  =  Increased  Greatly. 


New  Features 


Figure  31.  Mean  ratings  of  Effectiveness  of  New  Features  Across  Duty  Positions. 

Scale  Values:  1  =  Very  Ineffective,  3  =  Neutral,  5  =  Very  Effective 

New  Prototype  Features  Effectiveness.  Figure  31  provides  a  summary  of  the  findings 
on  effectiveness  of  the  new  C^  prototype  features  averaged  across  all  players  for  Run  5  and  Run 
10.  After  Run  5,  five  of  the  thirteen  new'  features  w'ere  rated  at  or  above  “Effective”  and  two 
features  were  rated  as  “Very  Effective”  (AGM  for  Threat  Index/Priority  and  Auto  Fires). 
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However,  the  HTR  Viewer  and  BDA  Recommendations  features  were  rated  below  “Neutral.” 
After  Run  10,  most  features  were  given  similar  ratings  by  the  players. 

Prototype  Support  of  Command  and  Control.  Figure  32  provides  summary  results  on 
the  average  effectiveness  of  the  prototype  in  support  key  command  and  control  tasks.  The 
results  indicated  that  the  prototype  effectively  supported  nearly  all  of  the  tasks  examined. 
However,  the  prototype  was  rated  as  less  than  “Effective”  for  Move  Ground  Assets,  Map  Display 
Manipulation,  and  particularly  Battle  Damage  Assessment. 


Figure  32.  Mean  ratings  of  Effectiveness  of  Prototype  Across  Duty  Positions  by  tasks. 

Scale  Values:  1  =  Very  Ineffective,  3  =  Neutral,  5  =  Very  Effective;  Run  8. 

Teamwork  Skills.  Selected  examples  of  effective  and  ineffective  teamwork  skills  based 

on  the  command  group  responses  to  the  Teamwork  Skills  questionnaire  are  provided  below: 

Communication: 

■  Effective:  Screen  sharing  provides  clear  and  accurate  information  exchange. 

■  Ineffective:  Unsynchronized  air  and  ground  recoimaissance. 

Coordination: 

■  Effective:  Team  coordinates  work  with  sjoichronization  matrix,  AGM,  group  overwatch  and 
follow.  The  UGS  fired  by  Effects,  but  locations  determined  by  Battlespace  Manager. 

■  Ineffective:  Focusing  resources.  The  frontage  of  30  km  is  too  large.  We  spend  more  time 
reacting  to  the  enemy  than  making  him  react  to  us.  Some  auto  fires  are  not  coordinated  with 
manual  fires,  results  in  double  taps  of  targets. 

Performance  Monitoring  and  Feedback: 

■  Effective:  Commander  can  view  everything  a  team  member  is  doing  and  give  feedback. 
Good  cross  talk  within  the  C^  Vehicle  during  planning  and  execution.  Team  does  a  good  job 
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of  providing  information  to  Battlespace  Manager  on  targets  engaged  and  which  ones  were 
hit.  Team  needs  to  monitor  any  continued  movement  of  targets. 

■  Ineffective:  Problems  if  each  team  member  just  worries  about  his  task  in  isolation. 

Shared  Situational  Awareness: 

■  Effective:  Screen  sharing  provides  100%  correct  shared  situational  awareness. 

■  Ineffective:  Screens  can  crash.  Team  members  can  become  too  focused  on  one  particular 
area  of  the  battle. 

Cf  Decision  Making.  Selected  examples  of  command  and  control  decisions  made  and 
the  support  provided  by  prototype  based  on  the  command  group  responses  to  the  Decision 
Making  questionnaire  identified  are  provided  below: 

Commander: 

■  Decisions:  When  to  begin  the  attack  and  cross  line  of  departure  (LD),  when  and  where  to 
pursue  the  attack. 

■  Supporting  prototype  features:  SA  (situational  awareness)  map  display  is  the  key,  the 
visual  information  is  intuitive  and  informative,  also  Battlefield  Assistant  and  Task  Manager. 

Battlespace  Manager: 

■  Decisions:  Route  selection,  system  to  select  for  engagements,  BDA. 

■  Supporting  prototype  features:  Auto  Route  Generation,  Quick  Fire  button.  Round 
Recommendation,  Chaser  rounds,  HTR/Enemy  Intelligence  window. 

Information  Manager: 

■  Decisions:  Focus  the  intelligence  collection  plan,  template  every  position  and  COA. 

■  Supporting  C^  prototype  features:  Easy  to  task  sensor,  but  sensor  server  is  unreliable.  The 
C^  prototype  allows  for  easy  templating  throughout  planning  and  execution.  Need  toggle  to 
turn  off  template  on  other  screens  to  improve  actual  SA. 

Effects  Manager: 

■  Decisions:  Configuration  of  Netfires  load,  weapon  selection  for  targets  of  opportunity,  best 
system  to  engage  a  type  of  target  for  automated  fires,  redirect  LAMs  for  location  of  orbit  or 
attack  mode. 

■  Supporting  C^  prototype  features:  Netfires  window  boxes  allow  configuring  number  of 
rounds  for  PAM,  LAM,  UGS.  Recommended  Fires  and  AGM  help  decide  best  weapons  to 
pair  with  targets  at  long,  medium,  and  near  distances.  Right  mouse  click  on  LAM,  open 
Redirect  window,  click  on  target,  assign  attack  mission  or  new  LAM  orbit  point. 

Human  Systems  Integration.  Results  by  function  indicated  low  workload  on  basic  C^ 
functions,  with  all  functions  rated  on  the  average  near  the  level  of  “Low  Workload”  (“3”  on  the 
workload  scale).  The  See  function  was  rated  highest  in  workload  (Mean  =  3.4),  then  Strike 
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(Mean  =  3.1),  Move  (Mean  =  2.6)  and  Plan  (Mean  =  2.3).  How'ever,  results  by  task  indicated 
higher  workload  for  several  tasks  where  the  average  rating  ranged  from  “Task  Attention 
Compromised  (“4”)  to  “Reduced  Attention  (“5”).  These  higher  workload  tasks  included  three 
See  related  tasks:  Interpreting  Sensor  Data,  Collecting  Picture  Data,  and  Interpreting  Picture 
Data;  and  two  Strike  tasks:  Designate  Target  and  Battle  Damage  Assessment. 

Results  on  prototype  usability  were  generally  positive.  Battle  Damage  Assessment  was 
the  only  task  that  was  ranked  by  the  majority  of  the  players  as  being  both  illogical/inconsistent, 
and  taking  too  many  steps  to  complete.  Most  tasks  were  rated  “Easy”  or  “Somewhat  Easy”  to 
perform.  The  three  tasks  rated  as  “Borderline”  by  at  least  one  player  with  respect  to  ease  of 
performance  were:  Visualizing  Past  and  Future  Threat  Positions,  Visualizing  Missile  Trajectory 
and  Intended  Target,  and  Determining  What  Entity  Detected  the  Threat  Target. 

Table  1 7  provides  a  summary  of  the  results  on  default  settings  for  the  C^  prototype. 
Player  feedback  in  the  form  of  “No  Change”  and  “Recommended  Change”  are  included  in  this 
table  for  the  seven  default  settings  listed  in  the  questionnaire  and  for  the  “Other”  category.  This 
feedback  should  help  designers  improve  the  default  settings  of  the  C^  prototype. 

Table  17 

Recommendations  for  changes  to  CSE/C^  Prototype  default  settings 


Default 

Duty  Position 

Recommended  Change 

Search  Radius  (LAM/PAM) 

No  Change 

Should  out  of  range  ammunition  be  an  option? 

No  Change 

When  out  of  a  tvpe  of  ammunition,  should  it  continue  to  be  the  default  or  even  an  option? 

Effects  Manager 

Would  be  nice  to  see  more  than  one  recommended  fire  option. 

1  Should  friendly  entities  be  a  default  for  weapons?  _  _ 

Battlespace  Manager 

Should  not  be  an  option  in  any  targeting  cue. 

Commander 

(No  written  recommendation  for  improvement) 

Should  picture  adjustments  be  saved  for  later  viewing  by  you  or  a  different  person?  _ . :: : _ _ J 

Commander 

Yes 

Battlespace  Manager 

Different  person. 

Information  Manager 

(No  written  recommendation  for  improvement) 

Should  your  tailored  workstation 

be  saved  when  the  system  crashes? 

Commander 

Yes  . - . 

Battlespace  Manager 

Yes  _ _ 

Information  Manager 

Yes,  first  30  seconds  used  to  redraw  map  and  settings. 

Should  craohics  such  as  the  NAIs  and  phaselines  be  click/drag  active  during  execution  phase?  _  | 

Commander 

Yes 

Battlespace 

Lock  them  down!  When  they  move  during  execution,  it  is  a  pain. 

Others 

Information  Manager: 

Need  to  put  the  toggles  back  for  enemy  units  (by  state). 

Need  to  add  toggles  for  weapons  (PAM,  LAM,  hits). 

Need  different  j^aphics  for  screen  clarity. 

Note:  If  a  command  group  player  is  omitted  under  a  question,  he  recommended  no  change  to  the 
default  setting. 
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Training.  Training  at  the  individual  level  was  substantially  improved  for  Experiment  3. 
A  formal  training  program  was  developed  and  provided  for  individual  training.  This  training 
was  intentionally  not  duty  position  specific,  but  rather  designed  to  provide  the  cross-duty  skills 
required  to  operate  the  prototype  for  any  of  the  four  primary  command  group  duty  positions. 
In  addition,  administrative  guidance  “fenced”  the  training  room  to  minimize  disruptions  from 
visitors  and  observers.  Observation  of  the  training  sessions  was  supported  by  teleporting  the 
sessions  to  other  rooms,  particularly  the  AAR  room.  Training  shortcomings  included  inadequate 
collective  training  and  a  neglect  of  plan  training.  Training  issues  and  recommendations  are 
examined  in  the  Conclusions  section. 


Conclusions 

This  section  provides  concluding  assessments  concerning  the  method  and  results  related 
to  Human  Functions  Assessment  for  Experiment  3  of  the  FCS  C^  program.  First,  conclusions  are 
provided  on  the  results  and  measurement  methods  for  Verbal  Interactions  and  Human-Computer 
Interactions.  Next,  more  general  conclusions  that  include  results  on  Subjective  Measures  as  well 
as  the  results  on  objective  verbal  and  HCI  behaviors  are  provided  on  the  following  topics: 
workload,  training,  automated  measures,  and  human-system  integration.  In  closing,  a  brief  set  of 
sustain  and  improve  recommendations  are  provided  for  future  research  efforts. 

Verbal  Interactions 

Several  method  limitations  with  respect  to  the  analysis  of  verbal  communications  for 
Experiment  1  and  2  were  identified,  and  at  least  partially  addressed  for  Experiment  3.  Method 
refinements  included  the  addition  of  two  additional  categories  for  characterizing  verbal 
communications:  Valence  and  Command  Considerations.  More  successful  method  refinemeiits 
from  Experiment  2  were  used  again  in  Experiment  3.  Particularly,  commimications  were  coded 
into  much  smaller  chunks  to  better  ensure  that  each  chunk  more  precisely  represented  a  unitary 
and  discrete  communication.  As  a  result,  the  range  of  inter-rater  agreement  achieved  in  coding 
verbal  communications  from  Experiment  2,  ranging  from  a  low  of  .86  for  the  Factor  category  to 
highs  of  .99  for  Source,  was  noteworthy.  A  similar  approach  to  chunking  was  used  for 
Experiment  3.  High  inter-rater  agreement  suggests  the  verbal  communication  coding  scheme 
provided  a  reliable  method  for  assessing  command  and  control  communications  that  might  be 
useful  in  a  wider  range  of  FCS  research  and  development  efforts. 

Overall,  the  verbal  data  from  the  execution  phase  produced  patterns  generally  consistent 
with  Experiments  1  and  2.  The  relative  distribution  of  communications  by  Source,  Function, 
Type  and  METT-TC  Factor  were  highly  similar  across  Experiments  2  and  3.  Verbal  data  from 
the  planning  phase  provided  some  interesting  differences  in  the  pattern  of  communication  during 
execution.  Despite  the  preliminary  nature  of  the  planning  data,  verbalizations  from  the  planning 
phase  provide  useful  indicators  for  gauging  execution  performance  and  improving  the  training  of 
future  command  groups. 

Verbal  results  from  the  execution  phases  provide  a  useful  basis  for  understanding 
command  and  control  processes  in  FCS  type  organizations.  Verbal  interaction  was  an  almost 
continuous  activity  during  each  of  the  execution  phases  analyzed  for  Experiment  3,  as  it  was  for 


59 


Experiment  2.  This  pattern  of  steady  verbalization  occurred  in  the  context  of  (a)  the  command 
group’s  access  to  a  visually  rich  and  timely  battlefield  depiction  on  their  prototypes,  and  (b) 
ongoing  interactions  with  their  prototypes  to  command  and  control  a  predominantly  robotic 
force.  A  concluding  hypothesis  is  that  the  level  of  verbal  communication  in  a  command  and 
control  organization  increases  as  the  level  of  visually  based  data/information  increases. 

Clearly,  the  verbal  communications  of  the  command  group  serve  multiple  functions, 
including  the  functions  documented  in  the  Results  section.  Most  importantly,  verbalization 
appears  to  be  the  command  group’s  method  of  choice  for  transforming  data  into  information. 
Although  future  systems  are  expected  to  help  commanders  visualize  the  battlefield,  verbal 
communications  may  still  be  required  to  collaboratively  process,  filter  and  interpret  the  data 
visually  displayed  into  meaningful  information.  The  steady  flow  of  verbalizations  underscores 
its  importance  for  command  and  control  coordination  and  collaboration  even  when  advanced 
multi-mode  technologies  are  employed.  In  sum,  the  command  group’s  vocalized  efforts  to 
transform  data  into  information  mirrors  the  evolving  doctrine  of  PCS  that  states  battle  command 
must  transform  “See  First”  into  “Understand  First.” 

More  specific  functions  of  verbal  communications  were  indicated  by  the  dominance  of 
Share  and  Ask  vocalizations.  Share  and  Ask  communication,  respectively  disclosed  the 
collaborative  nature  of  the  command  group’s  work,  and  a  fair  degree  of  uncertainty  about  their 
upcoming  battlefield  situations.  Moreover,  almost  80%  of  Enemy  related  communications 
concerned  Location  or  BDA,  with  approximately  equal  amounts  of  discussion  devoted  to  each. 
This  pattern  of  Enemy  related  communications  underscored  the  command  group’s  collaborative 
and  balanced  efforts  to  “find  and  fix”  Enemy  elements. 

Valence  ratings  helped  capture  the  affective  meaning  conveyed  by  command  group 
communications,  more  specifically  positive  and  negative  status  information  on  the 
accomplishment  of  functions.  Status  information  is  essential  to  maintaining  command  and 
control  in  dynamic  situations,  and  a  basis  for  evaluating  alternatives  and  making  decisions. 
Valence  also  highlights  problems  that  require  attention.  Negatively  valenced  communications 
provide  useful  diagnostic  information  on  the  limitations  of  the  prototwe  and  the  operational 
setting.  Efforts  by  the  FCS  program’s  Technical  Team  to  refine  the  d  prototype  and 
operational  environment  should  attend  to  the  problems  the  command  group  was  experiencing  as 
negatively  rated  verbalizations  occurred. 

Command  Considerations  were  introduced  for  Experiment  3  to  provide  a  more  explicit 
link  between  thought  and  action,  particularly  action  in  the  form  of  the  command  group’s  verbal 
interaction.  How  do  the  verbal  behaviors  of  the  participants  relate  to  the  cognitive  processes 
required  for  battle  command?  As  expected,  given  the  experienced  and  expert  command  group 
participants,  communications  related  to  all  nine  Command  Considerations  were  found.  Two  near 
exceptions  were  the  low  frequencies  tabulated  for  Coordinate  (create  synergistic  effects)  and 
Assets  (use  all  assets  available).  However,  with  experienced  command  groups,  the  coordinated 
use  of  all  assets  may  be  more  implied  than  explicit.  To  the  extent  that  the  Command 
Considerations  identified  are  key  to  battle  command,  comparisons  with  less  experienced 
participants  may  prove  more  discriminating  and  useful.  The  ARI  notes  that  the  Comprehensive 
Report  by  ARI  across  FCS  C^  Experiments  1-4  will  include  a  section  on  less  experienced 
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command  groups  based  on  the  FCS  Summer  Study  with  Cadets.  Command  Considerations 
may  also  prove  useful  in  setting  “baseline”  expectations  for  command  group  training. 

Results  on  planning  provide  useful,  if  preliminary,  data  on  the  important  but  largely 
neglected  topic  of  planning  for  small  unit  FCS  command  and  control.  The  ARI’s  decision  to 
focus  on  planning  was  based  on  two  primary  considerations,  as  noted.  The  evolving  concept  of 
FCS  command  and  control  stresses  the  requirement  for  more  effective  and  collaborative  planning 
methods  before  and  during  execution.  By  design,  the  planning  process  and  products  prior  to 
execution  provide  the  basis  for  planning  changes  during  execution,  sometimes  referred  to  as 
dynamic  re-planning.  Planning  for  a  manned  and  robotic  force,  such  as  the  Unit  Cell,  requires 
substantial  changes  in  the  planning  process  and  products  required  for  execution. 

The  preliminary  nature  of  the  planning  results  was  stressed  due  to  method  shortcomings. 
Even  the  findings  of  reduced  verbalizations  during  planning  was  due,  in  part,  to  player  absence 
from  the  C^  vehicle  and  failure  to  wear  headsets  consistently  during  the  planning  phase.  Notably, 
the  most  interesting  explanation  for  less  talk  may  be  actual  differences  in  the  communication 
requirements  for  planning  versus  execution.  However,  method  shortcomings  should  be 
eliminated  or  at  least  reduced,  before  more  firm  conclusions  on  actual  differences  are  reached. 

Verbal  interactions  disclosed  that  Plan  related  topics  were  the  most  frequently  discussed 
C^Function.  This  result  was  not  surprising,  but  quite  different  from  execution  communications. 
Similarly,  many  planning  communications  related  to  the  Mission  factor  in  METT-TC,  indicating 
the  players’  concerns  with  mission  goals  and  plans  prior  to  execution.  As  during  execution, 

Share  and  Ask  type  communications  dominated  planning  indicating  the  collaborative  nature  of 
the  command  group’s  work,  and  uncertainty  about  the  upcoming  battlefield  situation. 

A  “hot”  topic  during  planning  was  the  prototype  (IT/CSE),  particularly  concerns  and 
questions  about  how  to  correctly  input  operational  plans.  Players  expressed  uncertainty  about 
what  could  be  done  with  the  prototype  during  planning  versus  during  execution.  Given  Runs 
10  and  1 1  were  the  last  runs  during  Experiment  3,  this  topic  of  communication  suggests  that 
training  was  inadequate,  particularly  in  support  of  planning. 

In  sum,  to  improve  the  value  of  the  planning  data  obtained,  ARI  recommends  that  future 
efforts  modify  the  experimental  design  to  overcome  the  data  collection  shortcomings  and 
curtailed  planning  activities  noted  above.  However,  refinements  to  training  and  human 
performance  assessment  are  also  needed  to  better  see,  understand  and  improve  small  command 
group  planning  for  FCS. 

Human-Computer  Interactions 

Overall,  many  of  the  research  goals  for  HCI  analysis  were  at  least  partially  achieved. 
Based  on  ARI’s  understanding  of  the  new  features  developed  in  the  prototype  for  Experiment 
3,  and  a  review  of  all  Run  10  video  recordings,  the  HCI  rating  scheme  developed  for  Experiment 
2  was  revised  and  expanded  (see  Figure  1).  The  evaluation  framework  for  HCI  remained 
structured  to  basic  C^  Functions  (Plan,  See,  Move,  Strike)  and  Sub-Functions.  However,  the 
number  of  HCI  tasks  identified  increased  from  53  to  83  to  reflect  the  new  features  added  to  the 
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prototype  for  Experiment  3,  and  ARI’s  focus  on  documenting  the  HCI  tasks  performed  during 
planning  as  well  as  execution. 

Results  identified  new  HCI  tasks,  task  allocation  arrangements  across  the  command 
group,  and  even  task  allocation  across  the  dual  screens  provided  at  individual  workstations.  In 
general,  the  results  provided  quantitative  estimates  of  command  group  workload  and  the  impact 
of  automation  on  key  functions  and  tasks  during  execution  and  planning.  Overall,  the  results 
provide  an  emerging  empirical  basis  for  reallocation  of  tasks  among  the  command  group,  and 
between  the  command  group  and  increasingly  automated  systems. 

2 

Results  provided  precise  indicators  of  needed  improvements  in  the  design  of  future  C 
systems  to  minimize  workload  and  training  requirements.  For  example,  the  procedure  of  taking 
multiple  UAV  “snapshot”  pictures  around  templated  targets  yielded  hundreds  of  useless  target 
images  that  had  to  be  reviewed.  Similarly,  automated  features  such  as  “auto  recon”  and  route 
generation  may  relieve  players  of  the  routine  task  of  entering  route  points.  However,  these  same 
features  raise  the  demand  on  players  to  more  fully  understand  the  decision  rules  and  parameters 
designed  into  such  features.  During  Run  10,  the  Shadow  UAV  “wandered  off’  toward  enemy 
elements  under  auto  recon  and  was  inadvertently  destroyed,  a  critical  loss  to  the  See  First 
capability  of  the  Unit  Cell.  This  unintended  consequence  of  high  automation  was  due  to 
information  overload,  too  many  useless  images  to  view,  and  the  lack  of  an  effective  human 
override  to  abort  an  automated  routine  in  a  timely  manner. 

Results  from  planning  were  at  least  partially  successful.  Particularly  in  identifying  and 
documenting  the  frequency  and  duration  of  new  tasks  unique  to  planning  with  advanced  C 
prototypes  and  semi-automated  forces.  Planning  for  manned  and  robotic  forces  requires 
substantial  change  in  the  planning  process  and  products  required  for  execution.  The  results 
reported  provide  an  interesting  and  important  window  for  imderstanding  the  impact  of  FCS  on 
the  planning  process  of  small  command  groups. 

Again,  ARI  stresses  that  results  on  plaiming  are  incomplete  and  preliminary.  Method 
recommendations  were  made  for  overcoming  many  of  the  planning  shortcomings  identified. 

First,  the  introduction  of  new  missions  on  unfamiliar  terrain  would  require  more  extensive 
planning  and  provide  a  more  valid  base  for  planning  assessment.  Second,  more  representative 
play  of  upper  and  lower  echelons  would  raise  the  requirement  for  more  formal  and  tractable 
planning  procedures  and  products. 

Workload 

The  higher  levels  of  automation  provided  by  a  host  of  new  features  in  the  prototype  for 
Experiment  3  were  designed  to  at  least  partially  automate  key  functions,  particularly  See, 
Move,  and  Strike.  Many  of  these  new  features  were  well  received  by  the  participants.  By  their 
own  account,  the  new  automation  features  reduced  workload  for  some  tasks  and  increased  the 
effectiveness  of  the  command  group  and  the  Unit  Cell.  New  features  consistently  rated  as  work 
reducers,  included  the  Attack  Guidance  Matrix,  New  Alerts  and  Quick  Fire  features. 


62 


Automation  is  a  double-edged  sword,  however,  particularly  during  the  early  development 
cycle  that  characterized  the  new  prototype  features.  At  times,  automation  failed  to  perform  as 
expected.  At  other  times,  automation  performed  in  unexpected  and  detrimental  ways.  All  too 
often,  lack  of  trust  in  automation  was  an  issue  that  seriously  compromised  human  and  unit 
performance. 

Across  the  series  of  FCS  experiments,  overall  average  workload  ratings  have  varied 

substantially:  Means  =  57.6, 61 .2  and  48.9  for  Experiments  1  -3,  respectively.  The  increase  in 
Experiment  2  workload  appeared  due  to  changes  made  in  prototype  functionality  and  task 
allocation  between  experiments,  as  discussed  in  ARI’s  Interim  Report  for  Experiment  2.  During 
Experiment  2,  the  command  group  players  were  required  to  perform  additional  tasks,  such  as 
Effects,  Human  Target  Recognition  (HTR),  and  Battle  Damage  Assessment  (BDA).  Concerns 
that  sensors  provided  unrealistically  good  information  during  Experiment  1,  resulted  in  sensor 
degradations  and  a  shift  in  the  allocation  of  HTR  and  BDA  tasks  to  the  command  group  players 
during  Experiment  2.  In  contrast,  the  See  and  Move  focus  of  Experiment  1,  allocated  Strike  and 
supporting  effects  tasks  largely  to  higher  echelons.  During  Experiment  3,  the  lower  levels  of 
perceived  workload  seem  to  reflect  the  higher  levels  of  automation  provided  by  new  prototype 
features  in  support  of  See,  Move,  and  Strike  functions. 

Notably,  multiple  reviews  of  same  images  during  Experiment  3  were  substantially  less 
than  during  Experiment  2.  Credit  for  this  workload  reduction  goes  in  large  to  design  changes  in 
the  C^  prototype  to  provide  an  audit  trail  on  images  reviewed  and  by  whom.  This  is  the  type  of 
information  processing  assistance  readily  performed  by  computers  to  improve  the  efficiency  and 
effectiveness  of  command  group  performance,  particularly  for  human  intensive  tasks  such  as 
HTR  and  BDA. 

Many  of  the  workload  problems  associated  Avith  automation  and  experienced  during 
Experiment  3  were  expected.  Such  problems  are  typical  with  advanced  technology  insertions  in 
high-risk  and  dynamic  operational  settings.  For  command  and  control,  automation  issues  and 
problems  of  particular  concern  include:  allocation,  authority,  autonomy  and  awareness 
(Lickteig,  et  al.,  2002).  Such  human  performance  issues  must  be  addressed  and  resolved  before 
FCS  concepts  are  transformed  into  viable  constructs  and  solutions. 

Training 

Training  at  the  individual  level  was  substantially  improved  for  Experiment  3.  A  formal 
training  program  was  developed  and  provided  for  individual  training.  This  training  was 
intentionally  not  duty  position  specific,  but  rather  designed  to  provide  the  cross-duty  skills 
required  to  operate  the  prototype  for  any  of  the  four  primary  command  group  duty  positions. 

In  addition,  administrators  “fenced”  the  training  room  to  minimize  disruptions  from  visitors  and 
observers.  Observation  of  the  training  sessions  was  supported  by  teleporting  the  sessions  to 
other  rooms,  particularly  the  AAR  room. 

During  Experiment  3,  automation  from  new  features  increased  the  training  challenge 
substantially  at  individual  and  collective  levels.  Examples  of  training  shortcomings  include  the 
command  group’s  tendency  to  “double  tap”  numerous  targets.  A  lack  of  familiarity  with  and 
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feedback  on  the  AGM  system  caused  players  to  engage  targets  manually  due  to  doubts  about 
when  and  if  targets  would  be  effectively  engaged  by  the  AGM.  Similarly,  a  lack  of  experience 
in  monitoring  multiple  air  assets  and  anticipating  auto  recon  consequences  resulted  in  loss  of 
Shadow  and  Micro-UAV  sensors. 

Training  shortcomings  included  inadequate  collective  training  and  neglect  of  planning. 
Three  training  improvements  are  recommended  for  collective  training:  (1)  structured,  practical 
exercises  directed  at  collaboration  within  the  Unit  Cell  and  with  other  echelons,  (2)  exercises 
embedded  in  operational  conditions,  and  (3)  exercises  directed  at  planning,  not  just  execution. 
Understanding  prototype  complexities  for  planning — especially  input  requirements  and 

consequences  for  automated  See,  Move  and  Strike  functions — ^requires  structured  training 
directed  specifically  on  their  use  with  clear  operational  feedback.  Refresher  training  on  “old” 
features  is  also  required.  As  indicated  by  comments  such  as  “Where  did  the  Intervisibility  button 
go?” 


A  final  conclusion  is  that  the  training  challenge  associated  with  more  advanced,  and 
particularly  automated,  technology  must  not  be  underestimated.  The  ARI’s  conclusion  is  that 
automation  will  increase  the  training  challenge  substantially  at  individual  and  collective  levels. 
ARI’s  concerns  about  training  issues  as  a  result  of  automation  apply  to  the  prototype  evolving 
under  the  PCS  program,  and  to  the  “objective”  system  that  will  be  fielded  for  PCS. 

Automated  Measures 

The  introduction  of  automated  measures  of  human-computer  interaction  during 
Experiment  3  sharpened  the  program’s  focus  on  human  performance  measurement.  Prior 
experiments  had  relied  too  heavily  on  subjective  measures  about  performance  including 
questionnaires  and  interviews  rather  than  direct  measures  o/performance,  particularly  for  human 
performance  assessment. 

As  a  result,  for  Experiments  1  and  2  ARI’s  objective  measures  of  human  performance 
were  based  primarily  on  manual  reduction  of  audio  and  video  behaviors  from  recorded  runs.  The 
video  records  were  used  to  conduct  a  relatively  laborious  analysis  of  human-computer 
interactions,  namely  command  group  participant  interactions  with  the  C  prototype,  as 
documented  in  ARI’s  Interim  Report  for  Experiment  2. 

Overall,  results  on  development  and  validation  of  automated  measures  were  meager  but 
promising.  Only  3  of  the  23  automated  measures  requested  by  ARI  for  HCI  analysis  were 
developed  for  Experiment  3.  Two  of  the  three  automated  measures  developed  for  Experiment  3 
were  at  least  partially  validated.  Discrepancies  identified  for  Images  Requested Wie wed  and 
Create  Ground/ Air  Route  should  be  resolved  with  collaboration  by  behavioral  and  technical 
experts.  Additional  development,  refinement  and  validation  are  clearly  needed  to  develop  a 
useful  set  of  automated  measures  to  support  training,  evaluation  and  system  design. 

2 

Automated  performance  measures  are  required  to  support  training,  evaluation  and  C 
system  design  (Unit  of  Action  Maneuver  Battle  Laboratory,  2002).  This  requirement  applies  to 
the  PCS  prototype,  and  to  all  future  systems.  Manual  video  data  reduction  of  command 
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and  control  performance  can  only  examine  a  fraction  of  the  data  potentially  available  from  C 
research  and  training  efforts.  In  contrast,  automated  logs  of  human-computer  interactions 
provide  efficient  and  effective  measures  of  command  and  control  performance. 

The  efficiency  of  automated  measures  equates  to  quick  and  inexpensive.  It  includes  the 
ability  to  adjust  the  range  and  selection  of  data  to  include  the  performance  of  any  or  all  users 

at  any  or  all  times  during  an  operational  exercise.  The  effectiveness  of  automated  measures 
equates  to  increased  scope  and  precision  in  the  collection  of  performance  data.  It  includes 
more  meaningful  measures  by  automatically  correlating  performance  with  the  battlefield 
situation  in  which  it  occurred. 

During  the  AARs  for  Experiments  2  and  3  logs  of  simulation  and  system  activity  were 
used  to  provide  immediate  feedback  on  issues  such  as  shots  and  kills.  However,  no  logs  on 
human  activity  were  available  despite  serious  concerns  about  workload  and  task  allocation.  One 
example  of  the  human  performance  data  needed  for  Experiment  2  AARs  was  workload  data  on 
image  manipulation.  How  much  time  was  spent  manipulating  picture  images?  How  many  times 
was  the  same  picture  opened  and  manipulated  by  different  people?  Automated  measures  of 
human-computer  interactions  could  readily  answer  these  and  many  other  human  performance 
issues  in  the  area  of  command  and  control. 

Overall,  automated  human  performance  data  can  address  the  question  “How  well  did  the 
command  group  perform  the  C?  function?”  as  opposed  to  simply  asking,  “Who  won  the  battle?” 
For  the  FCS  program’s  commander-centered  focus,  such  data  are  more  valuable  in  assessing 
the  prototype  system  than  loss-exchange  ratios  and  other  battle  outcome  measures.  Battle 
outcomes  are  contingent  on  manipulations  of  force  size  and  capability,  but  performance  is 
contingent  on  system  design  decisions. 

Human-System  Integration 

From  a  human  performance  perspective,  the  interface  for  FCS  will  be  a  primary  point, 
or  means,  of  interaction  between  commanders  and  Soldiers,  and  between  commanders/Soldiers 
and  robotic  entities.  One  criterion  for  assessing  the  adequacy  of  a  interface  is  that  it  enables 
thought  and  action  in  a  manner  the  user  chooses  (Rasmussen  &  Pejtersen,  1995).  It  should  also 
provide  a  common  operational  picture  and,  particularly  for  FCS,  a  common  mode  of  interaction 
for  command  emd  control  of  manned  and  robotic  forces. 

A  final  criterion  noted  here  is  that  a  C^  interface  must  ensure  automated  functions  are  not 
too  transparent  or  invisible,  as  they  were  at  times  during  Experiment  3.  The  phrase  “transparent 
to  the  user”  often  connotes  a  good  thing.  However,  the  user  must  also  be  able  to  render  visible 
the  underlying  processes  and  actions  associated  with  the  automated  features  provided  by  a  C^ 
interface.  If  not,  the  complexity  and  uncertainty  induced  by  mediating  layers  of  technology  can 
severely  restrict  the  ability  of  humans  to  detect  unexpected  system  difficulties  and  consequences, 
or  even  provide  informed  consent  (Olson  &  Sarter,  2001). 

Developing  the  “objective”  interface  required  for  FCS  is  a  key  challenge  that  will 
require  iterative  use  and  refinement.  A  preliminary  conclusion  by  ARI’s  research  team  for  FCS 
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is  that  this  program  is  developing  a  cutting-edge  interface  for  FCS.  At  present,  this 
interface  reflects  the  sustained  (nearly  two  year)  use  and  refinement  by  an  exceptional  set  of 
command  group  participants.  Effective  interface  design  requires  sustained  user-based  input,  not 
a  bunch  of  guys/gals  sitting  around  a  table  (BOGS  AT)  or  a  drive-by  user  jury. 

Admittedly,  much  interface  refinement  work  remains  that  should  be  performed  in  future 
efforts.  However,  key  features  that  distinguish  the  FCS  interface  from  many  other  fielded  and 
prototype  interfaces  are: 

■  Value  grounded  on  sustained  use  and  refinement  by  expert  users. 

■  Provides  an  increasingly  effective  interface  for  command  and  control  of  robotic  entities. 

■  Integrates  of  human  and  robotic  forces  versus  “swivel-chair”  integration. 

Recommendations  for  Future  FCS  Experiments 

The  research  environment  and  iterative  experimental  design  of  the  FCS  C  program 
affords  an  excellent  venue  for  exploring  new  approaches  to  command  and  control.  The  human 
performance  findings  from  Experiment  1-3  serve  as  important  benchmarks,  and  the  lessons 
learned  provide  direction  for  future  FCS  efforts.  However,  method  improvements  are  needed  to 
help  meet  the  Army’s  future  command  and  control  requirements. 

The  ultimate  value  of  a  research  and  development  program  is  determined  as  much  by  the 
resources  spent  on  evaluation,  as  the  resources  spent  on  simulation.  And  the  ultimate  value  of  a 
C^  system  is  determined  not  by  the  technology  per  se,  but  by  shaping  technology  to  complement 
human  performance.  Some  key  sustain  and  improve  recommendations  are  bulleted  below  for 
future  FCS  C^  experiments: 

■  Maintain  Medium,  High,  and  Too  High  Complexity  levels  to  determine  performance  limits. 

■  Maintain  Deliberate  Practice  design  to  ground  findings  on  proficient  performance. 

■  Capitalize  on  the  requirement  for  and  value  of  automated  measures  of  human  performance. 

■  Add  novel  missions  and  terrain  to  broaden  the  spectrum  of  operations,  including  planning. 

■  Improve  methods  to  develop  and  disseminate  findings  across  doctrine,  organizations, 
training,  materiel,  leader  development,  personnel,  and  facilities  (DOTMLPF). 

■  Improve  methods  for  identifying  and  codifying  new  approaches  to  command  and  control  by 
documenting  the  C^  lessons  learned  during  After  Action  Reviews. 

■  Ensure  technology  complements  human  performance. 
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Appendix  A 
List  of  Acronyms 


AAR 

After  Action  Review 

ADA 

Air  Defense  Artillery 

AID 

Automatie  Target  Detection 

AFRU 

Armored  Forces  Research  Unit 

AGM 

Attack  Guidance  Matrix 

ARI-Knox 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  at  Fort  Knox 

ARL 

Army  Research  Laboratory 

AQ 

Area  Query 

BDA 

Battle  Damage  Assessment 

BOGSAT 

Bunch  of  Guys/Gals  Sitting  Around  a  Table 

& 

Command  and  Control 

&V 

Command  and  Control  Vehicle 

&1SR 

Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance 

CECOM 

U.S.  Army  Communications-Electronics  Command 

CEP 

Circular  Error  Probable 

COA 

Course  of  Action 

CSE 

Commander’s  Support  Environment 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DCSOPS&T 

Deputy  Chief  of  Staff  for  Operations  and  Training 

DOTMLPF 

Doctrine,  Organizations,  Training,  Materiel,  Leader  Development. 
Personnel,  and  Facilities 

DVO 

Direct  Vision  Optics 

FBC 

Future  Battlefield  Conditions 

FCS 

Future  Combat  Systems 

FIFV 

Future  Infantry  Fighting  Vehicle 

GCM 

Graphic  Control  Measure 

GSR 

Ground  Surveillance  Radar 

HCI 

Human-Computer  Interaction 

HUD 

Heads  Up  Display 

HTR 

Human  Target  Recognition 

IFV 

Infantry  Fighting  Vehicle 

IPT 

Integrated  Product  Team 

IR 

Infra  Red 
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ISR 

Intelligence,  Surveillance,  Reconnaissance 

IT/CSE 

Information  Technology/Commander’s  Support  Environment 

lUGS 

Intemetted  Unattended  Ground  Service 

LAM 

Loitering  Attack  Missile/Munitions 

LD 

Line  of  Departure 

LOS 

Line  of  Sight 

METT-TC 

Mission,  Enemy,  Terrain,  Troops  Available — ^Time  and  Civilians 

MOP 

Measure  of  Performance 

MPERM 

Multipurpose  Extended  Range  Munition 

MTI 

Moving  Target  Indicator 

NAI 

Named  Areas  of  Interest 

NASA 

National  Aeronautics  and  Space  Administration 

Netfire 

Network  Fire 

NLOS 

Non  Line  of  Sight 

NIC 

National  Training  Center 

OPCON 

Operational  Control 

PAM 

Precision  Attack  Missile 

PM 

Program  Manager 

RDEC 

Research  and  Development  Center 

ROTC 

Reserve  Officers’  Training  Corps 

SA 

Situation  Awareness 

SAR 

Synthetic  Aperture  Radar 

SD 

Standard  Deviation 

STO 

Science  and  Technology  Objective 

TK 

Track 

TRADOC 

Training  and  Doctrine  Command 

TQ 

Target  Query 

TTP 

Tactics  Techniques  and  Procedures 

TLX 

Task  Load  Index 

UAV 

Unmanned  Aerial  Vehicle 

UGS 

Unattended  Ground  Sensor 
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Appendix  B 

Verbal  Communication  Rating  Scheme:  Definitions  and  Examples 
For  each  chunk  select 


SOURCE  (for  each  verbal  chunk  select  one  and  only  one  Source  code) 


ll 

Within  Cell  (Black) 

Cell  =  4  C^  prototype  operators. 

m 

Cell  <->  Blue  (Team) 

Cell  <->  White  (Higher) 

Cell<->Subordinate 

Subordinate  (includes  C^ Vehicle  gunner  &  driver). 

B 

Blue<->  White 

More  than  2-way  (e.g., 
Cell<->White<->B  lue) 

Only  to  be  used  in  cases  where  more  than  2  elements 
involved  in  SAME  conversation. 

B 

Other 

E.g.,  to  technical  support  people. 

FUh 

fCTION  (for  each  verbal  chunk  select  one  and  only  one  Function  code) 

1 

See 

Detect  or  identify  enemy  or  fiiendly  positions,  or 
significant  terrain  aspects  (not  BDA). 

2 

Plan 

Interpret  data,  predict  enemy  CO  A,  generate  own 
COA 

B 

Move 

Manage/monitor/control  asset  movement. 

B 

Strike 

Manage/monitor/control  lethal/nonlethal  effects. 

5 

BDA 

See  for  purposes  of  BDA. 

6 

Other 

None  of  the  above. 

VALENCE  (for  each  verbal  chuck  so 

ect  one  and  only  one  Valence  code) 

1 

0 

-1 

See 

Ability  to  See 

Neutral/inconclusive 

Inability  to  See 

Plan 

Plan  Working 

Neutral/inconclusive 

Move 

Ability  to  Move 

Neutral/inconclusive 

Inability  to  Move 

Strike 

Ability  to  Strike 

Neutral/inconclusive 

Inability  to  Strike 

BDA 

Ability  to  Confirm  Kill 

Neutral/inconclusive 

Inability  to  Confirm 

Kill 

Other 

Other  Function 

Achieved 

Neutral/inconclusive 

Other  Function  not 
Allowed 
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TYPE  (for  each  verbal  chunk  select  one  and  only  one  Type  code) 


1 

Share.  Verbalization  about  what  is  seen  or  known. 

2 

Action.  Verbalization  about  what  speaker  is  doing  at  the  moment,  verbalization 
with  action  such  as  fire  or  move.  (Not  the  decision  process.  Not  actions  as  I  see, 
monitor,  track,  etc.  Not  describing  someone  else’s  actions.) 

3 

Direction.  Order,  command,  delegation  of  responsibility. 

4 

Ask.  Verbalization  begins  with  request  for  information,  confirmation,  assistance, 
or  assets  and  ends  with  either  informational  answer  or  no  response,  with  little  or  no 
discussion.  Not  rhetorical  questions. 

5 

Process.  Infer,  synthesize,  fuse,  understand,  and  turn  data  into  information  without 
consequent  decision  or  direction.  Can  start  with  Share,  Action,  or  Ask. 

6 

Decide.  Like  Process,  but  in  addition,  includes  a  verbalized  decision  or  plan. 

7 

Other. 

FACTOR  (for  each  verbal  chunk  select  one  and  only  one  Factor  code) 


MISSION 

1 

Original  Plan:  Concerning  mission  goals  and  plans  prior  to  execute  phase. 

2 

Dynamic  Plaiming:  Tactical  re-planning  during  the  execute  phase  in  response  to 
changing  events  and  available  assets.  Must  have  stated  COA  (course  of  action). 
Changes  from  Original  Plan. 

3 

Situational  Understanding.  Integration/summary  of  current  situation  involving 
multinle  factors,  but  without  stated  COA. 

EN 

EMY: 

4 

Location:  Sensor  hit(s)  -  locate  enemy  positions. 

5 

Identification:  Identify  targets  -  identify  nature  of  enemy  target. 

6 

Disoosition:  Probable  enemy  COA,  strategy,  or  tactics. 

7 

BDA:  Battle  Damage  Assessment  -  cell  seeks/discusses  feedback  on  damage  they 
inflict  on  enemy. 

TEl 

RRAIN 

8 

When  terrain  is  the  prime  focus  (e.g..  Can  we  travel  over  that  kind  of  terrain?  We 
should  go  this  way  because  it  will  provide  cover).  Example:  “Moving  to  low 
ground.”  Not  simply  map  locations  (e.g.,  not,  sensor  hit  north  of  the  wall). 

TROOPS  and  Assets  (Soldiers,  Equipment,  Vehicles) 

Friendly  only. 

9 

Location  Status:  Position  report/assessment. 

1 

0 

Movement  Status:  Mobility  report/assessment  (includes  fuel). 

1 

1 

See  Status:  Sensor  report/assessment. 

1 

2 

Strike  Status:  Firepower  report/assessment  (includes  #  of  remaining  missiles) 

1 

3 

Communications/network  functionality  (radio,  internet,  or  other;  cell  to  outside  cell, 
including  semi-autonomous  sensors). 

1 

4 

Information  management  systems:  &  prototype  user  interface  tools. 
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1 

5 

Survivability  Concern:  Asset  in  danger. 

1 

6 

Survivability  Move:  defensive  move  to  remove  asset  from  immediate  danger. 

1 

7 

Loss/Casualty:  Asset  destroyed  (catastrophic  hit). 

1 

Move  Action:  Move/Manage/Maneuver  [Active,  position  report] 

8 

Excluding  Survivability  Move;  Also  See  Terrain. 

1 

9 

Strike  Action  Lethal:  Launch/fire/deploy  with  intent  to  destroy  (includes  LAMs) 

2 

Strike  Action  Nonlethal:  Launch/fire/deploy  (could  include  unarmed  sensors, 

0 

propaganda,  smoke,  jamming  of  enemy,  etc.). 

2 

1 

Training  (Soldier  training,  mission  rehearsal). 

2 

2 

Other-  having  to  do  with  troops  or  assets  but  none  of  the  above. 

TIME 

2 

When  time  is  the  prime  focus  (e.g.,  how  much  time  something  will  take,  how  much 

3 

time  is  available,  order  of  priority,  synchronization  of  actions). 

CIVILIANS 

2 

Any  issues  regarding  how  to  deal  with  civilians:  avoiding,  provisioning,  protecting, 

4 

etc.  Not  mere  sensor  hits  of  civilians,  unless  first  time  mentioned. 

Other 

2 

5 

Other  (e.g.,  humor,  personal,  leadership,  morale). 
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Coding  rules  of  thumb  for  Verbal  Communication  Rating  Scheme: 


Type: 

Rationale:  Share,  Action,  and  Direction  are  meant  to  be  relatively  short  interactions, 
without  a  lot  of  discussion.  Chunks  including  a  lot  of  discussion  or  consideration  of  multiple 
aspects  of  situation  should  be  either  Process  or  Decide.  These  are  distinguished  by  whether  there 
is  a  definite  conclusion  reached  (Decide)  or  not  (Process). 

1 .  When  in  doubt  between  Share  and  Action,  choose  Share. 

2.  When  in  doubt  between  Share  and  Process  or  Decide  choose  Process  or  Decide  (as 
appropriate). 

3.  Mien  in  doubt  between  Ask  and  Process  or  Decide  choose  Process  or  Decide  (as 
appropriate). 

4.  Rhetorical  questions  or  questions  following  an  announcement  should  not  be  coded  as  Ask, 
(e.g..  I  have  a  mover,  do  you  see  it? — should  be  Share). 

5.  You  have  to  pay  attention  to  both  the  beginning  and  end  of  a  chunk.  If  contains  a  verbalized 
decision,  plan,  or  direction,  it  is  Direction  or  Decide,  regardless  of  how  it  begins.  Distinguish 
Direction  and  Decide  by  whether  it  is  preceded  by  some  discussion  (Direction  is  not; 
Direction  stands  alone.  Decision  is  preceded  by  relevant  discussion).  For  example,  a  sensor 
hit  followed  by  a  direction  to  fire  should  be  classed  as  a(type:  direction/subject:  lethal  effects, 
not  type:  share/subject:  sensor  hit.  (as  per  example  below). 

■  We’ve  ID  the  other  mover  wheel  coming  out  of  hidden  valley  is  a  URL. 

■  Dave  take  that  URL  with  a  PAM. 

Subject: 

Rationale:  Choose  the  major  subject  of  the  chunk.  Consider,  what  information  is  the 
speaker  trying  to  convey? 

1 .  Choose  Dynamic  Replanning  or  Situation  Understanding  when  the  conversation  contains 
discussion  of  multiple  assets.  Use  Dynamic  Replanning  when  it  does  include  a  course  of 
action  (here’s  what  we  should  do).  Use  Situation  Understanding  when  it  does  not  include  a 
course  of  action  or  plan,  but  only  summarized  the  current  situation.  If  chunk  contains 
discussion  of  only  a  single  asset,  choose  the  appropriate  category  related  to  that  type  of  asset 
or  action. 

2.  Sometimes  judgment  will  be  required.  In  these  cases  try  to  imagine  what  is  the  subject  the 
speaker  is  trying  to  convey?  (e.g.,  “Darya  found  by  Roboscout”  Context  will  usually  help.  I 
would  tend  to  code  this  as  Sensor  hit,  especially  if  it  is  the  first  time  the  hit  was  mentioned. 

On  the  other  hand,  a  preceding  question  regarding  which  sensor  system  detected  the  Darya, 
would  make  the  main  information  conveyed  by  the  utterance  Sensors. . .”  Who  found  the 
Darya?  Darya  found  by  Roboscout”). 

3.  When  the  Type  is  scored  as  “Ask,”  and  there  is  ambiguity  as  to  Subject,  focus  on  what  kind 
of  information  the  asker  is  after. 
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System: 

1 .  If  more  than  one  asset  is  mentioned,  give  more  than  one  system  code.  Note  the  existence  of 
categories  such  as  Other,  Unspecified,  or  Not  Applicable. 

■  Other — asset  is  mentioned,  but  is  not  one  of  the  choices. 

■  Unspecified — clearly  talking  about  one  of  the  assets  but  you  can’t  tell  which  (e.g.,  you  know 
it  is  a  lethal  effect,  but  you  don’t  know  which  one). 

Not  Applicable — a  system  is  simply  not  applicable  to  the  subject  discussed  in  the  chunk. 
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Appendix  C 

Examples  of  Chunks  from  the  10  Most  Frequent  Source-Type-Factor  Profiles 


Source 

Type 

Factor 

%  of  Run  11 

Within-Cell 

2.7 

6.6 

Example: 

There’s  one  Garm  there.  Okay,  I’ve  got  it. ... 

One  NL2,  and  a  combined  arms  platoon  at  460. 

Within-Cell 

Share 

Enemy  Disposition 

1.3 

4.9 

Example: 

Cause  if  we’re  going  to  have  a  combined  armed  reserves  out  here. 

Right. 

And  that’s  fine. 

Actually,  though.  Just  for  future  reference,  a  tank  platoon  in  his  order 
of  the  battle  is  three  platoons.  A  tank  platoon  in  our  area  of  course, 
is  four  tanks. 

Yeah,  is  four  tanks,  okay. 

I’ll  put  them  in  right  dovra  here.  This  is  where  I  think  they’ll  be. 

Alright,  that’s  good. 

Alright,  so  that  means  that  he’s  got  a  company  down  there,  and  a 
company  up  yonder.  Yonder. 

Within-Cell 

Share 

Prototype 

2.7 

4.9 

Example: 

Just  screwed  up...NAIs.  Uh-oh. 

The  one  all  the  way  up  there? 

Yeah.  No,  no. 

Delete  that  one.  I  don’t  need  that  one.  I  don’t  need  those  up  there.  I 
was  going  to  delete  them.  I  was  just  reassembling  them  around. 

There  we  go.  Is  there  anything  else  you  would  like  for  me  to  delete  up 
top? 

I  was  going  to  get  those.  I  can  get  those?  You  can  take  those  Garms  if 
you  want  to. 

Alright.  I’ve  got  them.  You  want  more? 

Nah,  that  should  work.  Just  enough  to  get  us  a  picture. 

Within-Cell 

Share  Other  0 

6.6 

Example: 

And,  how  are  we  doing,  guys?  Gals?  Anybody?  Women?  Children?  I 
say — You  know  what  I  say?  I  say  that  we  just  throw  the  game  on, 
whether  the  reds  are  ready  or  not.  It  should  be  like  a  surprise 
attack. 

Within-Cell 

Ask 

Mission* 

6.7 

4.9 

Example: 

We  got  two  Purga  locations? 

Ok  we  got  2  Purga  locations  there  in  the  Northeast.  Right  down  there  I 
put  PAMs  on  each  one  of  those  then  I  followed  them  up  with 

LAMs  to  see  if  we  can  get  some  movement.  I  am  reluctant  to  put 
more  PAMs  out  there  because  of  the...  I’ve  got  additional  LAMS. 
Check. 

Roger. 
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Within-Cell 

Ask  Enemy  Location 

5.3 

3.3 

Example: 

Tank  alpha  22  is  a  tank? 

The  very  bottom  of  the  plot? 

Check. 

He's  confirmed,  they  gave  us  6  digit  grids,  so  I  am  sure  there  is 
something  there. 

Within-Cell 

Process 

Enemy  Disposition 

5.3 

0 

Example: 

He  thinks  we  are  coming  there,  he  obviously  thinks  we  are  coming  in 
the  south 

Within-Cell 

Decide 

Mission 

6.7 

0 

Example: 

Then  that  means  we  have  to  make  a  conscious  effort  to  use  the 
Intelligence,  Surveillance,  Reconnaissance  (ISR)  Assets. 

Oh,  absolutely. 

Not  shoot  everything  in  the  North. 

Correct,  I  agree. 

Let  him  start  moving,  let  him  come  to  us. 

Cell-Blue 

Ask 

Communications 

1.3 

4.9 

Example: 

And,  Blue  6.  Black  6.  Radio  check.  Over.  Blue  6.  Black  6.  Radio 
check.  Over. 

Other 

Other 

Other 

0 

4.9 

Example: 

How  you  doing? 

Are  you  Jack? 

Yes,  I  am. 

This  is  Paul. 

Hey,  Paul.  How  you  doing? 

Total 

32.0 

41.0 
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Appendix  D 

HCI  Rating  Scheme  with  Description  of  Rating  Categories 

100  PLAN  (Describe  Tactical  Situation,  Concerns,  and  Future  Activities, 

Request  Information.) 

110.  CreateAJpdate  a  Mission  and  COA 

111.  Create  Overlay  Graphics  and  Map  Annotations 

112.  Place  platforms  on  the  map  (friendly  and  threat  template) 

113.  Rehearse  the  Plan 

1 14.  Execute  the  Plan 

115.  Point  on  Map  Using  Cursor/Indicate  an  Area 

1 16.  Move  icons  (vehicle  or  GCM)  on  map  using  drag/drop 

1 1 7.  Modify  overlay  graphics 

120.  Alerts 

121.  Create  Alerts 

200  MOVE  (Manage/Monitor  Control  Asset  Movement) 

210.  Move  Ground  Assets  (Start  =  First  blue  line  appear,  Stop  =  Click  OK) 

211.  Create  routes  (clicking  map  locations  to  create  blue  route  line) 

212.  Start,  Halt  or  Resume  a  platform 

213.  Edit  an  existing  route 

214.  Delete  all  tasks  (from  execution  window) 

215.  Place  UGS 

216.  Overwatch 

217.  Generate  Route 

218.  Recon  an  Area 

220.  Move  Air  Assets  (Start  =  First  blue  line  appear,  Stop  =  Click  OK) 

22 1 .  Create  Routes  (either  by  creating  a  direct  route  or  by  selecting  targets  to  recon) 

222.  Delete  all  tasks  (from  execution  window) 

223.  Edit  an  existing  route 

224.  Recon  an  Area/Auto  Recon  Targets 

225.  Task  to  Hover 

230.  Group  Follow 

23 1 .  Create  Ground  Follow 

232.  Create  Air  Follow 

233.  Create  Mixed  (Ground  and  Air)  Follow 

300  SEE  (Manage  Map  and  Sensor  Data  Display) 

310.  Manipulate  Map 

311.  Zoom  Map  (either  arrow  or  magnification  tools) 

312.  Scroll  Map 

320.  Use  Visualization  Aids 

321.  T oggle  Range  F ans 

322.  Plot  Intervisibility 

323.  Measure  Distance 

324.  Display  on  Heads  Up 

325.  Select/Change  Windows,  State  View,  or  window  area  for  display 
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(increasing/decreasing  a  window  area  such  as  Asset  or  Alert  window  for  better 
viewing) 

326.  Change  GCM  Settings  (Declutter  Map  -  Includes  Hide  Impacted  Missiles) 

327.  Move  Visual  Reference  Points  (red  cross  used  by  higher.  Expect  to  eventually 
include  in  analysis  all  who  manipulate  information  during  runs  on  the 
prototype) 

330.  Display  Sensor  Data 

33 1 .  Display  Target  Catalog  (open  this  spreadsheet  window,  is  it  normally  open?) 

332.  Query  Enemy  (cursor  over  enemy  icons  to  read  properties  information) 

333.  Query  Friendly  (cursor  over  friendly  icons  to  read  properties  information) 

334.  Query  Area  (cursor  over  area  to  read  properties  information) 

335.  Change  Sensor  (UAV,  Shadow,  Roboscout  etc.) 

336.  Toggle  sensor  fans 

337.  Acknowledge  Alert  Window 

338.  Highlight  Target  on  Map  using  Target  Catalog 

339.  Take  Manual  UAV  Picture 

340.  Recognize  Targets  (Start  =  HTR  window  open.  Stop  =  Close  HTR  window) 

34 1 .  Display  target  images  (through  Alert  Window,  clicking  the  picture  icon  on  map, 
select  window,  etc.) 

342.  Refine  image  (zoom,  pan,  brightness,  etc.) 

343.  Change  Map  Icons  to  reflect  target  status  (i.e.,  Garm,  Draega,  Bus,  etc.) 

344.  Remove  templated  targets  (State  View  selection) 

345.  Select  recon  target  by  clicking  icon 

346.  Select  recon  target  by  select  window 

350.  Assess  Battle  Damage  (Start  =  HTR  window  open.  Stop  =  Close  HTR  window) 

351.  Display  target  images  (through  Alert  window,  clicking  the  picture  icon  on  map, 
select  window,  etc.) 

352.  Refine  image  (zoom,  pan,  brightness,  etc.) 

353.  Change  Map  Icons  to  reflect  target  status  (i.e.  targeted,  suspected,  dead,  etc. 

354.  Assign  unit  to  BDA  through  BDA  Recommendations 

355.  Acknowledge  Alert  (Ground  hit/Unit  hit) 

360.  Summarize  Situational  Awareness 

36 1 .  Open  Threat  Management  Window 

362.  Query  Enemy  (Unit  Viewer) 

363.  Track  Unit  (Unit  Viewer) 

364.  Query  Enemy  (Unit  Viewer) 

365.  Update/Change  information  displayed  in  Battlefield  Assistant  (e.g.,  EFire,  FFire) 
400  STRIKE  (Distribute  Indirect  and  Direct  Effects  Over  a  Set  of  Targets) 

410.  New  Features 

411.  Recommend  Fire  Unit 

412.  Open  Quick  Fire 

413.  Engage  from  Enemy  Intel  Window 

420.  Target  Designation 

42 1 .  Designate  by  Icon  Click 

422.  Designate  by  Menu  Selection 

423.  Designate  by  “Select”  Window 

430.  Fire  Weapon  System 
43 1 .  Fire  Netfire  LAM 


432.  Reassign  LAM  Final  Attack  Command  (right  click  on  LAM  icon  and  reassign  to  attack) 

433.  Fire  Netfire  PAM 

434.  Fire  LOS 

435.  Fire  C^Vehicle  (Gun  and  Javelin) 

436.  Fire  FW  CARRIER  (IFV) 

437.  Fire  Dismount  Javelin 

438.  Delete  all  scheduled  fire  tasks 

440.  Monitor  Fires  Execution 

441 .  Query  LAM  (cursor  over  LAM  icons  to  read  properties  information) 

442.  Query  PAM  (cursor  over  PAM  icons  to  read  properties  information) 

450.  Scheduled  Fires 

451.  Set  Minutes  to  F ire 

452.  Set  Delimiters 

460.  Attack  Guidance  Matrix 

461.  Create  AGM 

462.  Select  target  category 

463.  Select  Autofire  threat  level 

464.  Set  Weapon  Priorities 

465.  Edit  Roles 

466.  Adjust  threat  index  settings 

500  OTHER  MANUAL  ACTS 


510  General 

511.  Reboot  system  (Start  =  Fatal  Error,  Stop  =  C^  Prototype  full  restart) 
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Appendix  E 


Example  of  Human-Computer  Interaction  Rating  Codes 
(Effects  Manager’s  Right  Screen  Rim  10) 


Duty  Position:  Effects  Screen:  4  Right  Rim:  10 


Run  Time  (minutes) 

Code 

Description 

Start 

Time 

Stop 

Time 

Typical  configuration  of  screen  =  50%  map,  25  %  asset 
window,  and  25  %  execution  window. 

0  00  25 

325 

Increase  Map  Window 

0  00  30 

312 

Scroll  Map 

0  00  57 

325 

Increase  Map  Window 

0  01  08 

312 

Scroll  Map 

0  01  22 

311 

Zoom  Map 

0  01  26 

312 

Scroll  Map 

0  01  38 

333 

Friendly  Query  (FQ) 

0  19  45 

312 

Scroll  Map 

0  19  49 

312 

Scroll  Map 

019  53 

334 

Area  Query  (AQ) 

0  19  56 

333 

FQ 

0  20  14 

421 

Select  Target  by  Clicking  Icon 

0  20  28 

433 

Fire  PAM 

0  24  18 

333 

FQ 

0  24  33 

423 

Select  target  by  “select”  window 

0  24  46 

433 

Fire  PAM 

0  25  00 

332 

Target  Query  (TQ) 

0  25  01 

334 

AQ  . 

0  26  42 

338 

Highlight  Enemy  by  Target  Window 

0  26  47 

312 

Scroll  Map 

1  04  11 

312 

Scroll  Map 

1  04  32 

332 

TQ 

IIEE^ 

332 

TQ 

1  04  50 

332 

TQ 

1  05  22 

412 

Quick  Fire  Open 

1  05  25 

421 

Select  target  by  clicking  icon 

1  05  28 

433 

Fire  PAM 

1  06  07 

332 

TQ 

lliRicIM 

441 

LQ 

441 

LQ 

ll»RiW 

442 

PQ 

1  06  47 

334 

AQ 

1  06  49 

312 

Scroll  Map 

1  06  54 

312 

Scroll  Map 
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Run  Time  ( 

minutes) 

Code 

Description 

Start 

Time 

Stop 

Time 

lEaEOl 

311 

Zoom  Map 

UklM 

311 

Zoom  Map 

1  09  37 

334 

AQ 

1  09  45 

311 

Zoom  Map 

1  09  47 

332 

TO 

441 

IQ _ _ _ 

■IMiM 

332 

TO 

fipni 

332 

TO 

412 

Ouick  Fire  Open 

IIEI^H 

421 

Select  target  by  clicking  icon 

1  17  20 

433 

Fire  LAM 

1  23  40 

End  of  Execution 
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Appendix  F 


Frequency  of  Human-Computer  Interactions  by  Function, 
Sub-Function,  HCI  Task  and  Duty  Position,  Run  10  Execution 


Function 

Sub-Function 

HCI  task 

Commander 

Freq. 

Battlespace 

Manager 

Freq. 

Information 

Manager 

Freq. 

Effects 

Manager 

Freq. 

Move 

— 

28 

27 

Move  Ground  Assets 

— 

28 

— 

— 

Create  routes 

— 

11 

— 

— 

Start,  halt  or  resume  a  platform 

— 

4 

— 

— 

Edit  and  existing  route 

— 

5 

— 

— 

Delete  all  tasks 

— 

2 

— 

— 

Create  Overwatch  Task 

— 

1 

— 

— 

Generate  Route 

— 

5 

— 

— 

Move  Air  Assets 

— 

— 

27 

— 

Create  routes 

.... 

— 

18 

— 

Delete  all  tasks 

— 

— 

1 

— 

Edit  an  existing  route 

— 

— 

5 

— 

Task  UAV  to  hover 

— 

— 

3 

— 

See 

101 

189 

304 

188 

Manipulate  Map 

11 

18 

6 

91 

Zoom  map 

2 

6 

5 

42 

Scroll  map 

9 

12 

1 

49 

Use  Visualization  Tools 

6 

29 

26 

12 

Toggle  Range  fans 

2 

1 

— 

— 

Measure  distance 

— 

2 

— 

— 

Display  heads  up 

1 

4 

6 

1 

Select/change  windows/state 
View 

2 

14 

16 

3 

Change  GCM  settings 

1 

8 

4 

8 

Display  Sensor  Data 

45 

66 

102 

62 

Display  target  catalog 

— 

1 

1 

— 

Query  enemy 

23 

20 

24 

41 

Query  friendly 

2 

22 

26 

7 

Query  area 

14 

14 

49 

13 

Change  sensor 

— 

5 

1 

— 

Toggle  sensor  fans 

6 

4 

1 

— 

Highlight  Target  by  Catalog 

— 

— 

— 

1 

Fvmction 

Sub-Function 

HCI  task 

Commander 

Battlespace 

Manager 

Information 

Manager 

Effects 

Manager 
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Recognize  Targets 

1 

2 

37 

1 

Display  target  images 

1 

— 

5 

Refine  image 

— 

— 

3 

Change  icons  to  reflect  type 

— 

— 

6 

Remove  templated  targets 

— 

— 

1 

1 

Recon  by  clicking  icon 

— 

— 

13 

- — 

Recon  by  select  window 

— 

2 

9 

— 

Assess  Battle  Damage* 

9 

42 

132 

4 

Display  target  images 

8 

24 

67 

4 

Refine  image 

1 

18 

61 

“““ 

Change  map  icons  to  reflect 
target  status 

— 

32 

4 

— 

Situation  Awareness 

29 

1 

18 

Open  threat  management 

1 

— 

Query  enemy  (Unit  Viewer) 

6 

14 

— 

11 

Track  unit  (Unit  Viewer) 

15 

1 

— 

2 

Query  friendly  (Unit  Viewer) 

7 

17 

— 

5 

Change  Information  Given 

— 

— 

1 

— 

Strike 

4 

90 

7 

103 

New  Features 

— 

1 

— 

19 

Open  Quick  Fire 

— 

1 

— 

19 

Designate  Target 

— 

41 

1 

28 

Designate  by  icon  click 

— 

9 

1 

21 

Designate  by  menu  selection 

— 

— 

, 

— 

Designate  by  select  window 

— 

32 

— 

7 

Execute  Fires 

— 

46 

1 

32 

Fire  LAM 

— 

™ 

— 

3 

Reassign  Lam 

— 

— 

1 

6 

Fire  PAM 

— 

3 

— 

23 

Fire  LOS 

— 

40 

— 

— 

Fire  IFV 

— 

3 

— 

— 

Monitor  Fires 

4 

2 

5 

24 

Query  LAM 

4 

1 

2 

17 

Query  PAM 

— 

1 

3 

7 

Other 

— 

2 

— 

•*-**“ 

Reboot  System 

— 

2 

— 

— 

Total 

105 

309 

338 

291 
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Appendix  G 

Summary  of  Command  Group  HCI  Performance  Across  Time,  Run  10  Execution 

00-10  Players  concern  themselves  with  tailoring  their  workstations  (e.g.,  zoom  and  scroll),  AGM  makes 
most  fires  and  pre-plan  fires  emerge.  All  four  players  exhibit  concerns  about  situational 
awareness  (e.g.,  target,  area  and  friendly  queries)  and  perform  mostly  See  related  tasks. 

1 0-20  Players  continue  to  tailor  their  workstations.  Players  heighten  their  situational  awareness  of  the 
battlefield,  and  perform  many  tasks  related  to  battle  damage  assessment.  Few  fires  are  performed 
and  arial  sensors  are  moved  for  reconnaissance.  Most  tasks  performed  are  in  support  of  the  See 
function. 

20-30  Players  continue  to  tailor  their  workstations  and  continue  situational  awareness  at  a  decreased 
level.  Many  tasks  are  centered  around  battle  damage  assessment  and  an  increased  level  of  fires 
are  performed.  Most  tasks  performed  are  in  support  of  the  See  function.  No  movement  takes 
place  in  this  section. 

30-40  Players  greatly  reduce  their  tasks  related  to  tailoring  their  workstations.  Increased  situational 

awareness  occurs  during  this  time  period  as  well  as  a  decrease  in  executed  fires.  Many  tasks  are 
still  associated  with  battle  damage  assessment  and  the  majority  of  all  tasks  performed  are  related 
to  the  See  function. 

40-50  Ground  and  arial  movements  commence  at  a  higher  level  in  this  section  due  to  the  loss  of  the 
Shadow.  No  tasks  are  performed  in  relation  to  tailoring  of  the  workstation,  and  situational 
awareness  and  battle  damage  assessment  tasks  remain  at  the  previous  level.  Few  fires  are 
performed  as  most  tasks  are  related  to  the  See  function. 

50-60  Movement  continues  due  to  the  loss  of  the  Shadow,  and  players  display  an  increase  in  situational 
awareness  tasks.  Battle  damage  assessment  remains  a  priority  and  both  the  Battlespace  Manager 
and  Effects  Manager  greatly  increase  their  executed  fires.  A  majority  of  the  tasks  support  the  See 
function,  but  during  there  is  a  distinct  increase  in  Strike  tasks  performed  during  this  period. 

60-70  Movement  occurs  for  ground  units  toward  the  end  of  this  period  due  to  enemy  artillery  and 

reconnaissance.  Players  are  once  again  greatly  concerned  with  tailoring  their  workstations  and 
situational  awareness.  Many  tasks  are  performed  in  support  of  human  target  recognition  and 
battle  damage  assessment.  Fires  remain  at  a  heightened  level.  Most  tasks  are  performed  in 
support  of  the  See  function. 

70-80  Movement  continues  due  to  artillery  fire  and  reconnaissance.  Levels  of  tailoring  the  workstation, 
situational  awareness  and  battle  damage  assessment  remain  high.  Fires  are  again  increased  by  the 
Effects  Manager.  The  majority  of  all  tasks  support  the  See  function. 

80-84  During  the  final  stages  of  the  run,  players  mostly  concern  themselves  with  situational  awareness 
and  battle  damage  assessment  with  a  few  executed  fires.  Most  tasks  represent  the  See  function. 
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Appendix  H 
Subjective  Measures 


Workload  and  Performance  -  Task  Load  Index  Rating  Scales 

Prototype/CSE  Effectiveness  in  Support  of  Functions  and  METT-TC  Factors 
Skill  Proficiency 

Prototype/CSE  Features  Impact  on  Workload 
Prototype/CSE  New  Features  Effectiveness 
C^  Prototype/CSE  Effectiveness  in  Support  of  Command  and  Control 
C^  Teamwork  Skills 
C^  Decision  Making 

Human  Systems  Integration  Questionnaire 
Training 
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Task  Load  Index  Rating  Scales 

Task  or  Mission  Segment:  _ _ _ 

Please  rate  the  task  or  mission  segment  by  putting  a  mark  on  each  of  the  six  scales  at  the  point  which 
matches  your  experience. 


Mental  i  l  I  I 

Demand  I  I  I  I  I  I  I  I  I  I 

Very  Low 

(HOW  MENTALLY  DEMANDING  WAS  THE  TASK?) 


Very  High 


Physical 

Demand  |  |  |  |  I  I  I  I  II 

Very  Low 

(HOW  PHYSICALLY  DEMANDING  WAS  THE  TASK?) 


Very  High 


Temporal  i  i  i  i  l  I  I 

Demand  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 

Very  Low 

(HOW  HURRIED  OR  RUSHED  WAS  THE  PACE  OF  THE  TASK?) 


Very  High 


Performance  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I _ 

Failure  Perfect 

(HOW  SUCCESSFUL  WERE  YOU  IN  ACCOMPLISHING  WHAT  YOU  WERE 
ASKED  TO  DO?) 


Effort  MINIMI  I  M  M  I  M  M  I  I 

Very  Low  Very  High 

(HOW  HARD  DID  YOU  HAVE  TO  WORK  TO  ACCOMPLISH  YOUR  LEVEL  OF 
PERFORMANCE?) 


Frustration  11  I  I  11  11  I  I  I  11  I _ _ I _ 

Very  Low 

(HOW  DISCOURAGED,  IRRITATED  OR  ANNOYED  WERE  YOU?) 


Very  High 
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C2  FUNCTIONS  AND  METT-TC  SURVEY 


Part  2.  CSE  Effectiveness 

Duty  Position  _  Date _ Run  # 


Across  all  Runs,  how  effective  was  the  CSE  in  support  of  C^  Functions  and 
METT-TC  Factors? 


Functions 


■A«’ 


l.PLAN;CreateMission/COA,  tools  (range  fans,  intervisibility).  NA  1  2  3  4  5 

Comments:  _ 


2.  MOVE:  Create  routes,  move,  halt,  retask  ground  and  air  assets.  NA  1  2  3  4  5 

Comments: _ 


3.  SEE:  Target  data.  Human  Target  Recognition,  sensor  data).  NA  1  2  3  4  5 

Comments: _ 


4.  SHOOT:  Plan  and  Execute  fires,  check  resources,  BDA  NA  1  2  3  4  5 

Comments:  _ _ 


METT-TC  Factors 

1.  Mission:  Operations  Plan,  Dynamic  Planning,  Coord  w/ Higher  NA  1  2  3  4  5 

Comments: _ 


2.  Enemy:  Activities,  Composition,  Probable  COA  NA  1  2  3  4  5 

Comments: _ 


3.  Terrain:  Key  Terrain,  Avenues,  Observation  Lines,  Fields  of  fire.  NA  1  2  3  4  5 

Comments:  _ _ 


4.  Troops  &  Assets:  Training  adequacy.  Vehicles,  Sensors,  Weapons  NA  1  2  3  4  5 

Comments: _ 


5.  Time:  Maneuver,  Coord.,  Asset  task  time  (ex.  time  of  flight)  NA  1  2  3  4  5 

Comments:  _ 


6.  Civilians:  Identifying,  tracking,  avoiding,  protection.  NA  1  2  3  4  5 

Comments:  _ 
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AFTER  RUN  SURVEY 


Part  2.  Skill  Proficiency 

Duty  Position  _  Date _  Run  # _ 

At  this  time,  how  proficient  are  you  as  an  individual  and  as  a  collective  team? 


Proficiency 


1  2  3  4  5  6  7  8  9 

Extremely  Very  Low  Below  Average  Above  High  Very  Extremely 
Low  Low  Average  Average  High  High 

Individual 

1.  How  technically  proficient  are  you  at  using  the  CSE  to  perform  your  C2  Cell 
individual  tasks? 

Comments:  _ _ 


2.  How  tactically  proficient  are  you  at  serving  in  your  C2  Cell  duty  position. 
Comments:  _ _ 


Team 


1 .  How  technically  proficient  is  the  C2  Cell  Team  in  using  the  CSE  to  perform 
collective  tasks? 

Comments: _ _ _ _ _ 


2.  How  technically  proficient  is  the  C2  Cell  Team  in  directing  the  actions  of  robotic 
assets? 

Comments:  _ _ _ 


3.  How  tactically  proficient  is  the  C2  Cell  Team  in  communicating  (gathering/ 
sharing  information)? 

Comments:  _ _ _ 


4.  How  tactically  proficient  is  the  C2  Cell  Team  in  making  key  decisions? 
Comments:  _ _ 


Rating 

(1-9) 
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CSE  FEATURES  SURVEY 


Part  2.  New  CSE  Features  Workload 
Duty  Position  _  Date 


Run  # 


Across  all  Runs,  how  did  these  CSE  features  impact  overall  workload?^  ^ 

CSE  Automated  Features  ^  ^ 

1.  Enemy  Intel/Human  Target  Recognition  (HTR)  Viewer  NA  1  2  3  4  5 

Comments:  _ 


2.  Battle  Damage  Assessment  Recommendation  (recon)  NA  1  2  3  4  5 

Comments:  _ _ _ _ _ 


3.  Fire  Recommendation 
Comments:  _ 


NA  1  2  3  4  5 


4.  Attack  Guidance  Matrix  (AGM) 
Comments:  _ 


NA  1  2  3  4  5 


5.  Threat  Manager 
Comments: 


NA  1  2  3  4  5 


6.  New  alerts  (e.g.,  PAM/LAM  feedback,  self-protection)  NA  1  2  3  4  5 

Comments:  _ 


7.  Group  Tasking  (Overwatch,  Follow)  NA  1  2  3  4  5 

Comments:  _ 


8,  Quick  Fire  NA  1  2  3  4  5 

Comments:  _ 
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CSE  SURVEY 


Part  2.  CSE  New  Features  Effectiveness 

Duty  Position  _  Date _  Run  # 

Across  all  Runs,  how  effective  were  these  CSE  features? 


1 .  Enemy  Intel/HTR  Viewer 

Comments: 

NA 

2.  Icon  depiction  of  Engage  States/Fire  States 

Comments: 

NA 

3.  Icon  depiction  of  BDA  Indicators  (ties  with  Enemy  Intel  screen) 
Comments: 

NA 

4.  BDA  Recommendations  (recon) 

Comments: 

NA 

5.  Fire  Recommendations 

Comments: 

NA 

6.  Scheduled  Fire 

Comments: 

NA 

7.  Quick  Fire 

Comments: 

NA 

8.  Fire  from  HTR/BDA  Enemy  Intel  Screen 

Comments: 

NA 

9.  Attack  Guidance  Matrix  (AGM)  for  threat  index/priority 
Comments: 

NA 

10.  Attack  Guidance  Matrix  (AGM)  for  auto  fires 

Comments: 

NA 

1 1 .  New  alerts  (e.g.,  LAM/PAM  feedback,  self-protection) 

Comments: 

NA 

12.  Threat  Manager 

Comments: 

NA 

13.  Group  Tasking  (Overwatch,  Follow) 

NA 

Comments: 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

12  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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AFTER  RUN  SURVEY 


Part  2.  CSE  Support  of  C2  Tasks 

Duty  Position  _  Date _  Run  # 

How  effective  was  the  CSE  in  support  of  C2  tasks? 


1 .  Create/Update  a  Mission  and  COA  (Create  graphics,  rehearse  plan)  NA 
Comments: _ _ 


2.  Move  Ground  Assets  (Create  and  edit  routes)  NA 

Comments:  _ _ 


3.  Move  Air  Assets  (Create  and  edit  routes)  NA 

Comments: _ 


4.  Map  Display  Manipulation  (Zoom,  Scroll)  NA 

Comments: _ _ _ 


5.  Use  Visualization  Aids  (Range  fans,  head  up  display,  change  GCM)  NA 
Comments: _ 


6.  Sensor  Data  Display  (Create  alerts,  bring  up  target  properties)  NA 

Comments: _ _ 

7.  Human  Target  Recognition  (Display/refine  image,  change  icons)  NA 

Comments:  _ _  ' _ 

8.  Battle  Damage  Assessment  (Display/refine  image,  change  icons)  NA 

Comments: _ _ _ 

9.  Pre-Plan  Fires/Execute  Pre-Plan  Fires  (AGM,  Auto  fires)  NA 

Comments: _ _ _ 


10.  Fire  a  Weapon  System  (Netfires,  LOS,  C2V,  FW  Carrier,  Javelin)  NA 
Comments: _ 


1 1 .  Target  Designation  (Icon  click,  Menu  select)  NA 

Comments: _ 


12.  Monitor  Fires  Execution  (Cursor  over  LAM,  PAM)  NA 

Comments: _ 


•4^ 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 


1  2  3  4  5 
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AFTER  RUN  SURVEY 


Part  2.  C2  Teamwork  Skills 

Duty  Position  _ _  Date  _ Run#  _ 

Please  provide  an  example  of  Effective  and  Ineffective  performance  for  each  of  the  four 
C2  activities  listed  below. 

1 .  Communication:  Information  clearly  and  accurately  exchanged  between  team  members,  the 
ability  to  clarify  or  acknowledge  the  receipt  of  information. 

Effective _ _ _ _ _ 


Ineffective 


2.  Coordination:  Team  resources,  activities,  and  responses  are  organized  to  ensure  tasks  are 
integrated,  synchronized,  and  completed  within  established  time  constraints. 

Effective _ _ _ — _ 


Ineffective 


3.  Performance  monitoring  and  feedback:  Team  members  monitor  the  performance  of 
teammates,  give,  seek,  and  receive  task-clarifying  feedback,  and  offer  advice. 


Effective 


Ineffective 


4.  Shared  situational  awareness:  Team  members  share  a  common  model  of  the  team’s  internal 
and  external  environment,  maintain  a  conunon  understanding  of  the  situation,  apply  appropriate 
task  strategies. 

Effective _ _ _ _ _ 


Ineffective 
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AFTER  RUN  SURVEY 


Part  2.  C2  Decision  Making 

Duty  Position _  Date _  Run# _ 

Please  briefly  describe  several  IMPORTANT  DECISIONS  you  had  to  make  during  this 
Run,  and  identify  CSE  features  that  support  the  decision  ivhere  applicable. 


1 .  Decision: 


CSE  Features: 


2.  Decision: 


CSE  Features: 


3.  Decision: 


CSE  Features: 


4.  Decision: 


CSE  Features: 
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Name  _ _  Duty  Position  _  Date  _ 

Human  Systems  Integration  Questionnaire 

This  questionnaire  asks  for  your  feedback  on  how  well  the  CSE/C^  Prototype  system  supports 
human  information  requirements  for  battle  command. 

1 .  Using  the  workload  description  rating  scale,  provide  a  rating  of  your  perceived  workload  by 
writing  the  corresponding  number  in  the  “Rating”  column  for  each  of  the  foWovAngfunctions 
(plan,  move,  see,  strike)  and  related  tasks  from  1  to  10.  Please  add  any  other  key  tasks  that 
impact  workload  by  writing  them  in  the  cells  labeled  “Others?”  below. 

Workload 


1  2  3  4  5  6  7 

Insignificant  Very  Low  Task  Reduced  Little  High 

Low  Attention  Attention  Spare 

Compromised  Capacity 


8  9  10 

Very  Extremely  Task 

High  High  Abandoned 


Function  and  Task 

Rating 

(1-10) 

Recommended  Improvement  to  Reduce  Workload 

Plan 

Position  Icons  on  the  Map 

Rehearse  the  Plan 

Mission/Task  Change 

Others? 

Move 

Ground  Asset 

Air  Asset 

Position  Unmanned  Ground 
Sensors  (UGS) 

Task  Sensor  to  Recon 

Others? 
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WORKLOAD 


123456789  10 


Insignificant  Very  Low 
Low 


Function/Task 


See 


Task  Reduced  Little  High  Very  Extremely  Task 
Attention  Attention  Spare  High  High  Abandoned 

Compromised  _ Capacity _ 


Rating 

(1-10) 


Recommended  Improvement  to  Reduce  Workload 


Map  Manipulation  (eg.  Zoom) 


Using  Heads  Up  Display 


Altering  Workstation  Layout 


Creating  Alerts 
Collecting  Sensor  Data 
Interpreting  Sensor  Data 


Collecting  Picture  Data 


Interpreting  Picture  Data 
Others? 


Strike 


Detect  Targets 


Recognize  Targets 


Classify  Targets 


Identify  Targets 


Designate  Target 


FireNETFIRES 


Fire  Line  of  Sight  (LOS) 

Battle  Damage  Assessment 

(BDA) _ 

Others? 
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2.  Please  check  Yes  or  No  for  each  item  in  column  2a  (task  clarity)  and  2b  (task  complication) 
below.  _ _ _ _ _ 


a.  Are  there  any  tasks  that  are  not  logical  or 
consistent? 

b.  Did  any  tasks  require  too  many  steps  to 
complete? 

Share  information  on  Heads  Up  Display 

Yes  No 

Share  information  on  Heads  Up  Display 

Yes  No 

Create  routes 

Yes  No 

Create  routes 

Yes  No 

Edit  existing  tasks 

Yes  No 

Edit  existing  tasks 

Yes  No 

Measure  distance 

Yes  No 

Measure  distance 

Yes  No 

Tailor  workstation  (window  size  and  log  out) 

Yes  No 

Tailor  workstation  (window  size  and  log  out) 

Yes  No 

Changing  sensor  used 

Yes  No 

Changing  sensor  used 

Yes  No 

Human  Target  Recognition  (HTR) 

Yes  No 

Human  Target  Recognition  (HTR) 

Yes  No 

Target  designation 

Yes  No 

Target  designation 

Yes  No 

Allocating  search  radius  (LAM/PAM) 

Yes  No 

Allocating  search  radius  (LAM/PAM) 

Yes  No 

Fire  NETFIRES 

Yes  No 

Fire  NETFIRES 

Yes  No 

Fire  LOS 

Yes  No 

Fire  LOS 

Yes  No 

Fire  Javelin 

Yes  No 

Fire  Javelin 

Yes  No 

Battle  Damage  Assessment  (BDA) 

Yes  No 

Battle  Damage  Assessment  (BDA) 

Yes  No 

3.  Please  place  a  checkmark  in  the  appropriate  box  that  corresponds  to  the  amount  of  trouble 
you  experienced  viewing  the  following  display  characteristics. _ _ 


-  - ^ 

Display 

Characteristics 

o  . . . . b 

Never 

Have  Trouble 

_ A - C - 

Seldom 

Have  Trouble 

Occasionally 
Have  Trouble 

Frequently 
Have  Trouble 

Legibility  of  text 

Contrast  between 
symbols  and 
background 

Brightness  of 
displays 

Size  of  displays 

Color  of  symbols 

Text  on  displays 
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4.  Please  place  a  checkmark  in  the  appropriate  box  that  corresponds  to  the  amount  of 


difficulty/ease  in  accomj 

plishing  the  following  tasks. 

Symbology 

Characteristics 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Ease  of  distinguishing 
between  friendly  and 
threat  icons 

Ease  of  distinguishing 
between  moving  and 
stationary  threat  icons 

Ease  of  visualizing  past 
and  future  threat 
positions 

Ease  of  distinguishing 
between  LAM/PAM 
missile  icons 

Ease  of  visualizing 
missile  trajectory  and 
intended  target 

Ease  of  determining  what 
entity  detected  the  threat 
target 

Ease  of  understanding 
navigation  symbology 
(waypoints,  hazards,  etc.) 

5.  Are  there  any  menus  or  tasks  that  are  either  too  difficult  to  navigate  or  take  too  much  time 
that  they  complicate  your  performance?  Please  circle:  YES  NO 

Please  explain: 


6.  Are  there  any  items  in  the  CSE/  Prototype  improperly  or  confusingly  labeled? 
Please  circle:  YES  NO 

Please  explain: 
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7.  Please  place  a  checkmark  in  the  appropriate  box  that  corresponds  to  how  easy  is  it  to 

read/understand  the  data  provided  in  the  following  CSE/C^  Prototype  windows.  _ 

^VeryEas^^  Easy  ^^^BoirierIine^____^^^Difficul^^^^^^er^D^ 

Map  Window 

Mission 
Workspace 
(Table  of 
Organization  and 

Equipment) _ _ _ 

Execution 

Window 

(mission  timeline) _ 

Resource 

Availability 

Asset  Window 

Alert  Ticker 

Window  _ 

Alert  Editor 

Target  Catalog 

Battlefield 

Assistant 

Graphic  Control 
Measures  (GCM) 

Window _ 

Collection 

Management 

(CM)  Planner _ I _ 1 _ 
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8.  Are  there  any  CSE/C^  Prototype  default  settings  that  you  would  recommend  a  change?  If 
there  is  no  change  that  you  recommend  for  a  given  default,  please  place  a  check  mark  in  the 
“No  Change”  column  for  that  default.  The  following  blank  cells  are  for  your  suggestions  on 
other  defaults  that  need  adjustments.  _ 


Default 


No 

Change 


Recommended  Change 


Search  Radius  for  LAM/PAM 
Should  out  of  range  ammo  be  an  option 

When  out  of  a  type  of  ammo,  should  it 
continue  to  be  the  default  or  even  an  option 

Should  friendly  entities  be  a  default  for 
weapons  (RoboScout,  UGS) 

Should  picture  adjustments  be  saved  for  later 
viewing  by  you  or  a  different  person 

Should  your  tailored  settings  and  user 
preferences  be  saved  and  automatically 

loaded  when  the  system  crashes _ 

Graphics  such  as  the  Named  Area  of 
Interests  (NAIs)  and  phaselines  being 
click/drag  active  during  execution  phase 
Others? 
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Training  Survey 


Name  _ _  Duty  Position _  Date  _ 

Training  Adequacy 

How  adequate  was  the  individual  training  provided  to  prepare  you  for  your  duty  position  in  the 
C2  cell,  in  terms  of  content  coverage  and  time  spent? 

CONTENT;  _ _ _ _ _ 


TIME: 


How  adequate  was  the  collective  training  provided  to  prepare  the  members  of  the  C2  cell  to 
work  as  a  team,  in  terms  of  content  coverage  and  time  spent? 

CONTENT:  _ _ _ _ _ 


TIME: 
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